Jul 24 01:29:39.447333 ip-10-0-129-127 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 01:29:39.893328 ip-10-0-129-127 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 01:29:39.893328 ip-10-0-129-127 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 01:29:39.893328 ip-10-0-129-127 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 01:29:39.893328 ip-10-0-129-127 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 01:29:39.893328 ip-10-0-129-127 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 01:29:39.893328 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.889759    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894779    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894792    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894796    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894799    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894802    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894805    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894809    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894811    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894814    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894817    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894819    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894822    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894824    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894827    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894830    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894833    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894837    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894841    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 01:29:39.899471 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894843    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894846    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894849    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894857    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894859    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894862    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894865    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894868    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894870    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894873    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894876    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894879    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894882    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894884    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894887    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894890    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894893    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894895    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894898    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894901    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 01:29:39.924646 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894903    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894906    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894908    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894912    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894916    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894918    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894921    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894924    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894927    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894930    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894933    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894936    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894938    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894941    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894944    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894947    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894951    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894953    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894956    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 01:29:39.941251 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894959    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894962    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894965    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894967    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894970    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894973    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894976    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894978    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894981    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894984    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894986    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894989    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894991    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894994    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894996    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.894999    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895002    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895004    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895007    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895009    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 01:29:40.011237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895012    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 01:29:40.005058 ip-10-0-129-127 systemd[1]: Started Kubernetes Kubelet.
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895016    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895033    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895038    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895041    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895044    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895047    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895050    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895052    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895055    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895058    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895061    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895063    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895066    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895069    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895072    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895074    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895077    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895081    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 01:29:40.016160 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895084    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895087    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.895090    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897488    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897495    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897499    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897502    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897505    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897508    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897510    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897513    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897516    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897521    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897524    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897527    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897529    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897532    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897535    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897538    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897541    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 01:29:40.020787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897544    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897546    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897549    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897552    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897555    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897558    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897560    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897563    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897565    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897568    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897570    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897573    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897575    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897578    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897581    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897583    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897587    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897591    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897593    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897596    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 01:29:40.024959 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897599    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897602    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897605    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897608    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897613    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897616    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897618    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897621    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897623    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897626    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897630    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897633    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897635    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897638    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897641    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897644    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897646    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897649    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897651    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 01:29:40.028532 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897654    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897657    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897659    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897661    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897664    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897666    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897669    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897672    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897675    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897678    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897680    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897683    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897685    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897688    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897690    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897693    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897696    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897700    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897702    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897705    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 01:29:40.032673 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897708    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897710    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897713    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897716    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897720    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897723    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897727    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897730    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897748    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897752    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897757    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897760    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897763    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897766    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897769    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897772    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897774    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897777    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897780    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 01:29:40.038936 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897783    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897785    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897788    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.897790    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898210    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898216    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898219    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898222    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898228    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898231    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898235    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898238    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898242    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898244    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898247    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898250    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898253    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898256    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898259    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898261    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 01:29:40.197957 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898264    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898267    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898269    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898272    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898274    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898277    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898280    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898282    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898285    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898288    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898291    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898293    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898296    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898298    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898301    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898304    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898306    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898309    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898311    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898315    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 01:29:40.213806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898320    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898323    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898327    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898329    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898332    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898335    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898337    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898340    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898342    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898345    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898348    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898350    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898353    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898355    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898358    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898361    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898364    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898366    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898369    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898372    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 01:29:40.215385 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898374    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898376    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898379    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898381    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898384    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898386    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898389    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898391    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898394    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898396    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898399    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898401    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898405    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898408    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898412    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898415    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898417    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898420    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898423    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898425    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 01:29:40.218704 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898428    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898430    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898433    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898436    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898439    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898442    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898445    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898447    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898449    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898452    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898454    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898457    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898459    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898461    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898465    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898469    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898473    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898477    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898479    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 01:29:40.220623 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898482    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898484    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898487    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.898490    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898559    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898567    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898574    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898580    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898585    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898590    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898595    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898599    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898602    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898605    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898609    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898612    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898615    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898619    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898622    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898625    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898627    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898630    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898634    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898637    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 01:29:40.222944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898640    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898643    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898646    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898650    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898653    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898656    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898659    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898662    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898665    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898668    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898671    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898674    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898678    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898682    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898685    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898691    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898694    2564 flags.go:64] FLAG: --enable-server="true"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898697    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898701    2564 flags.go:64] FLAG: --event-burst="100"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898704    2564 flags.go:64] FLAG: --event-qps="50"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898707    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898710    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898713    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898717    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 01:29:40.225295 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898720    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898723    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898726    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898729    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898732    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898735    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898738    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898741    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898744    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898747    2564 flags.go:64] FLAG: --feature-gates=""
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898751    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898753    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898756    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898759    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898762    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898765    2564 flags.go:64] FLAG: --help="false"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898768    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-127.ec2.internal"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898771    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898774    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898777    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898780    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898786    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898789    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 01:29:40.226683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898793    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898796    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898799    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898801    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898805    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898808    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898811    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898813    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898817    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898819    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898822    2564 flags.go:64] FLAG: --lock-file=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898825    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898829    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898832    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898837    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898840    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898843    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898846    2564 flags.go:64] FLAG: --logging-format="text"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898848    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898852    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898854    2564 flags.go:64] FLAG: --manifest-url=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898857    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898862    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898865    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898869    2564 flags.go:64] FLAG: --max-pods="110"
Jul 24 01:29:40.228212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898871    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898875    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898878    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898881    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898884    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898891    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898894    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898903    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898907    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898910    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898913    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898916    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898922    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898925    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898928    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898930    2564 flags.go:64] FLAG: --port="10250"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898933    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898937    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0389c398d2d12a053"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898940    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898943    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898946    2564 flags.go:64] FLAG: --register-node="true"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898949    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898953    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898956    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 24 01:29:40.229680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898958    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898961    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898965    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898968    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898971    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898973    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898976    2564 flags.go:64] FLAG: --runonce="false"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898979    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898982    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898985    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898988    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898991    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898994    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.898999    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899002    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899005    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899009    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899012    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899014    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899017    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899020    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899036    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899042    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899044    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899047    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 01:29:40.230770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899051    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899054    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899057    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899060    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899064    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899067    2564 flags.go:64] FLAG: --v="2"
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899071    2564 flags.go:64] FLAG: --version="false"
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899075    2564 flags.go:64] FLAG: --vmodule=""
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899078    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.899082    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899190    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899193    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899196    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899199    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899202    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899204    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899207    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899210    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899213    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899216    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899220    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899223    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 01:29:40.231811 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899226    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899231    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899234    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899237    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899240    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899243    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899245    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899248    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899251    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899253    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899256    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899258    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899262    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899266    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899269    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899272    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899275    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899279    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899281    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 01:29:40.232848 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899284    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899287    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899289    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899292    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899294    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899297    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899301    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899304    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899307    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899309    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899312    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899315    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899318    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899320    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899324    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899327    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899330    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899332    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899335    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 01:29:40.237860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899338    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899340    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899343    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899345    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899348    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899351    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899353    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899356    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899358    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899361    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899364    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899367    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899369    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899372    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899374    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899377    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899380    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899383    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899385    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899388    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 01:29:40.238860 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899390    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899393    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899395    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899398    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899401    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899405    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899409    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899413    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899416    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899419    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899422    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899424    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899427    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899430    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899432    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899435    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899437    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899440    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899443    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899445    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 01:29:40.251176 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899448    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899451    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899453    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899456    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899459    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899461    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899464    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899466    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.899469    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.900437    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.907836    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.907851    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.907867    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.907873    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907933    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 01:29:40.251955 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907938    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907942    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907945    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907948    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907951    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907953    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907957    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907959    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907962    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907965    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907967    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907970    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907972    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907975    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907978    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907980    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907983    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907985    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907988    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907991    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 01:29:40.252767 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.907999    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908002    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908005    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908008    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908011    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908014    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908016    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908019    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908037    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908040    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908043    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908047    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908051    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908055    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908059    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908062    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908065    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908068    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908071    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 01:29:40.253559 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908075    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908078    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908080    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908083    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908086    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908088    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908091    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908094    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908097    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908099    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908102    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908105    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908108    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908111    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908114    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908117    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908120    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908122    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908125    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908128    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 01:29:40.254370 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908130    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908133    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908136    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908138    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908144    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908147    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908149    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908152    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908154    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908158    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908160    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908163    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908166    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908169    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908171    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908174    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908176    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908179    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908181    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908184    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 01:29:40.351171 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908187    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908190    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908194    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908198    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908201    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908204    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908207    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908211    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908213    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908217    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908220    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908223    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908225    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908228    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908231    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908234    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908238    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908240    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 01:29:40.351850 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908243    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.908249    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908375    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908380    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908382    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908385    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908388    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908391    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908394    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908397    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908399    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908402    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908405    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908407    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908427    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 01:29:40.352625 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908431    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908434    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908436    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908439    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908442    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908445    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908447    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908451    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908454    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908457    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908459    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908462    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908464    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908467    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908470    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908474    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908477    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908479    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908482    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908484    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 01:29:40.353057 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908487    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908490    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908492    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908495    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908497    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908500    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908502    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908505    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908507    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908510    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908513    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908515    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908518    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908520    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908523    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908526    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908531    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908533    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908536    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908538    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 01:29:40.353858 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908541    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908544    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908547    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908549    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908552    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908556    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908560    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908565    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908568    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908572    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908576    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908578    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908581    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908584    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908587    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908589    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908592    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908595    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908597    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 01:29:40.354661 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908600    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908602    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908605    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908607    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908609    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908612    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908614    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908617    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908619    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908623    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908626    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908629    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908631    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908634    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908637    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908640    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908643    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908646    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908648    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 01:29:40.355442 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908651    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908655    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908658    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908660    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908663    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908665    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908668    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:39.908670    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.908675    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.910557    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.913817    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.915685    2564 server.go:1019] "Starting client certificate rotation"
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.915780    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 01:29:40.356242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.915815    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.919760    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.921008    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.951529    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.959057    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.966376    2564 log.go:25] "Validated CRI v1 image API"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.967248    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.971173    2564 fs.go:135] Filesystem UUIDs: map[0bcd1fb2-422e-48ab-8a3a-a6794cbe5487:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9418d5dd-bfba-49ed-85a5-a0b6eed5718a:/dev/nvme0n1p3]
Jul 24 01:29:40.357052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.971189    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.977760    2564 manager.go:216] Machine: {Timestamp:2026-07-24 01:29:39.976333463 +0000 UTC m=+0.398400318 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110302 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af5ea1edc8e99bebe35bcf9793ad6 SystemUUID:ec2af5ea-1edc-8e99-bebe-35bcf9793ad6 BootID:56d53292-93a6-4591-8e05-aa78bda561c8 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:18:5e:a3:30:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:18:5e:a3:30:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:ee:2e:28:73:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.977857    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.977945    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.982541    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.982570    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.983427    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.983438    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.983445    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.984899    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.986104    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988070    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988200    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988233    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988252    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988268    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988279    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988292    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 01:29:40.357841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988758    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.988780    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.991195    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.994136    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.994175    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997062    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997081    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997090    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997098    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997110    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997119    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997131    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997138    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997149    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997156    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997167    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.997176    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.999331    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.999342    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:39.999971    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-czq2z"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.001388    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-czq2z"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.002302    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.002441    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.002508    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.004066    2564 server.go:1268] "Started kubelet"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.004174    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.004177    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.004268    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.005349    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.008718    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.012376    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.021997    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.022002    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.022128    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.021306    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-127.ec2.internal.18c5159a5a284cd9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-127.ec2.internal,UID:ip-10-0-129-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-127.ec2.internal,},FirstTimestamp:2026-07-24 01:29:40.004007129 +0000 UTC m=+0.426073958,LastTimestamp:2026-07-24 01:29:40.004007129 +0000 UTC m=+0.426073958,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-127.ec2.internal,}"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.022762    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.022808    2564 factory.go:55] Registering systemd factory
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.022826    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.022898    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.022910    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 01:29:40.358645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023020    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023046    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023052    2564 factory.go:153] Registering CRI-O factory
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023065    2564 factory.go:223] Registration of the crio container factory successfully
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023105    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023125    2564 factory.go:103] Registering Raw factory
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023137    2564 manager.go:1196] Started watching for new ooms in manager
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.023421    2564 manager.go:349] Starting recovery of all containers
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.032399    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.033865    2564 manager.go:354] Recovery completed
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.035550    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-127.ec2.internal\" not found" node="ip-10-0-129-127.ec2.internal"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.039331    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041156    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041184    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041195    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasSufficientPID"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041825    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041834    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041854    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.041879    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.042552    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.042577    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.042589    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasSufficientPID"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.122181    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.193403    2564 policy_none.go:49] "None policy: Start"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.193434    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.193451    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.194570    2564 policy_none.go:47] "Start"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.201803    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.203069    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.203103    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.203127    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.203171    2564 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:40.205569    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.222413    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:40.359862 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.303873    2564 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jul 24 01:29:40.594647 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.504312    2564 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jul 24 01:29:40.948916 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:40.904932    2564 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jul 24 01:29:41.716257 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:41.705463    2564 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jul 24 01:29:41.739577 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:41.738157    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:41.739577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.738193    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 01:24:40 +0000 UTC" deadline="2028-01-11 09:03:20.615367646 +0000 UTC"
Jul 24 01:29:41.739577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.738223    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12871h33m38.877148239s"
Jul 24 01:29:41.739577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.738229    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 01:29:41.847486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.752885    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 01:29:41.847486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.753016    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:41.847486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.768437    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:41.847486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:41.826160    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:41.847486 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:41.838216    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:41.964963 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:41.939149    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.039805 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.039783    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.062950 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.062925    2564 manager.go:337] "Starting Device Plugin manager"
Jul 24 01:29:42.063095 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.062988    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 01:29:42.063095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.063002    2564 server.go:87] "Starting device plugin registration server"
Jul 24 01:29:42.063335 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.063319    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 01:29:42.063403 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.063339    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 01:29:42.063449 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.063429    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 01:29:42.063538 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.063524    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 01:29:42.063538 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.063536    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 01:29:42.066094 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.066066    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 01:29:42.066205 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.066145    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.102265 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.102227    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:42.102265 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.102258    2564 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:42.141728 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.141703    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:42.156804 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.156788    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-127.ec2.internal" not found
Jul 24 01:29:42.164515 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.164486    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 01:29:42.165627 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.165610    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 01:29:42.165697 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.165642    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 01:29:42.165697 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.165655    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeHasSufficientPID"
Jul 24 01:29:42.165697 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.165685    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-127.ec2.internal"
Jul 24 01:29:42.175399 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.175383    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-127.ec2.internal"
Jul 24 01:29:42.175445 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.175405    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-127.ec2.internal\": node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.203320 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.203289    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.305065 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.304964    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.410384 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.410345    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.511665 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.511619    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.617172 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.617096    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.722593 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:42.722558    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-127.ec2.internal\" not found"
Jul 24 01:29:42.741640 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:42.741615    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 01:29:43.310813 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.310116    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"]
Jul 24 01:29:43.312314 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.312297    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.313407 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.313388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.329677 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.329660    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.333819 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.333805    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.341807 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.341794    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 01:29:43.344548 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.344533    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 01:29:43.351236 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.351216    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31744d3a57961580d2625e5c1e1dcc8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal\" (UID: \"31744d3a57961580d2625e5c1e1dcc8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.351287 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.351242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31744d3a57961580d2625e5c1e1dcc8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal\" (UID: \"31744d3a57961580d2625e5c1e1dcc8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.351287 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.351262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496a950db9b46b88536b00684a81f3f6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-127.ec2.internal\" (UID: \"496a950db9b46b88536b00684a81f3f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.452484 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.452437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31744d3a57961580d2625e5c1e1dcc8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal\" (UID: \"31744d3a57961580d2625e5c1e1dcc8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.452484 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.452483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31744d3a57961580d2625e5c1e1dcc8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal\" (UID: \"31744d3a57961580d2625e5c1e1dcc8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.452719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.452511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496a950db9b46b88536b00684a81f3f6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-127.ec2.internal\" (UID: \"496a950db9b46b88536b00684a81f3f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.452719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.452560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31744d3a57961580d2625e5c1e1dcc8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal\" (UID: \"31744d3a57961580d2625e5c1e1dcc8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.452719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.452574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496a950db9b46b88536b00684a81f3f6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-127.ec2.internal\" (UID: \"496a950db9b46b88536b00684a81f3f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.452719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.452589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31744d3a57961580d2625e5c1e1dcc8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal\" (UID: \"31744d3a57961580d2625e5c1e1dcc8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.645320 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.645229    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.647645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.647620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal"
Jul 24 01:29:43.739328 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.739292    2564 apiserver.go:52] "Watching apiserver"
Jul 24 01:29:43.739572 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.739298    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 01:29:43.749690 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.748422    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 01:29:43.749690 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.749411    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9kkdx","openshift-ovn-kubernetes/ovnkube-node-85shc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl","openshift-cluster-node-tuning-operator/tuned-fjqnz","openshift-dns/node-resolver-r8dkq","openshift-image-registry/node-ca-4dtpj","openshift-multus/multus-additional-cni-plugins-tj678","openshift-multus/network-metrics-daemon-crts2","openshift-network-operator/iptables-alerter-fdjrl","kube-system/global-pull-secret-syncer-2dzp8","kube-system/konnectivity-agent-dhr7p","kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal","openshift-multus/multus-vlpg6"]
Jul 24 01:29:43.751352 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.751335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.752481 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.752454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.753551 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.753527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgtgz\""
Jul 24 01:29:43.753551 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.753511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:43.753733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.753618    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.753733 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.753641    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:43.753733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.753655    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.754483 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.754447    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.754483 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.754477    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t44w7\""
Jul 24 01:29:43.754606 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.754590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.754705 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.754688    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.754754 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.754713    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 01:29:43.755633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.755620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.756790 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.756774    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.757197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757172    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 01:29:43.757323 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757309    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-thcns\""
Jul 24 01:29:43.757361 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757337    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.757652 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757637    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.757743 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757725    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.757900 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757856    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 01:29:43.757952 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.757856    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 01:29:43.758130 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.758112    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.758248 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.758233    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.758364 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.758307    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2m47\""
Jul 24 01:29:43.758364 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.758330    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 01:29:43.759346 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.759331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:43.759438 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.759392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:43.759497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.759457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.759497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.759457    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 01:29:43.759497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.759480    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qxrd4\""
Jul 24 01:29:43.759757 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.759721    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.760172 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.760158    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 01:29:43.760438 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.760425    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.760661 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.760646    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82jpg\""
Jul 24 01:29:43.760706 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.760664    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.760976 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.760963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.762564 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.762548    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.762634 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.762616    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:43.762931 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.762915    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mhpgb\""
Jul 24 01:29:43.764223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764047    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 01:29:43.764223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764068    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 01:29:43.764223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764116    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 01:29:43.764223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764203    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 01:29:43.764479 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764278    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 01:29:43.764479 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764394    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.764803 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.764785    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 01:29:43.766243 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.766226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.766660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.766641    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 01:29:43.766746 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.766730    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d8b67\""
Jul 24 01:29:43.766976 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.766959    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 01:29:43.768210 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.768196    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 01:29:43.768746 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.768726    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cbtqp\""
Jul 24 01:29:43.780644 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.780620    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sv68"
Jul 24 01:29:43.788678 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.788660    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sv68"
Jul 24 01:29:43.823848 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.823830    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 01:29:43.855325 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.855414 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-kubernetes\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.855414 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-sys\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.855414 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrt99\" (UniqueName: \"kubernetes.io/projected/25e37d82-6241-46f1-974c-655e53ab2189-kube-api-access-jrt99\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.855504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzhm\" (UniqueName: \"kubernetes.io/projected/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-kube-api-access-bfzhm\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.855504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-ovn\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.855504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-system-cni-dir\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.855504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855508    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-system-cni-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-cnibin\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-serviceca\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855571    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-systemd\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855587    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eeead595-ea5f-4a2e-a731-bb8a56c7702a-konnectivity-ca\") pod \"konnectivity-agent-dhr7p\" (UID: \"eeead595-ea5f-4a2e-a731-bb8a56c7702a\") " pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-os-release\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.855633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-hostroot\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2h2t\" (UniqueName: \"kubernetes.io/projected/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-kube-api-access-d2h2t\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-device-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-dbus\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-host\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-registration-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855805    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-run-netns\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-cni-bin\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.855850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-conf-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cnibin\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-slash\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-var-lib-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-etc-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-log-socket\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-ovnkube-config\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-host\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.855991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-systemd-units\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-etc-kubernetes\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-var-lib-kubelet\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31b8df2f-e374-4426-b777-905a07afac34-ovn-node-metrics-cert\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-iptables-alerter-script\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-run\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-run-ovn-kubernetes\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-cni-multus\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856197    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkjb\" (UniqueName: \"kubernetes.io/projected/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-kube-api-access-slkjb\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-modprobe-d\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-lib-modules\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25e37d82-6241-46f1-974c-655e53ab2189-hosts-file\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-sys-fs\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-node-log\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-env-overrides\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8rq\" (UniqueName: \"kubernetes.io/projected/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-kube-api-access-xb8rq\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-kubelet\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-ovnkube-script-lib\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-socket-dir-parent\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874nk\" (UniqueName: \"kubernetes.io/projected/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-kube-api-access-874nk\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856502    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-kubelet-config\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.856611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856517    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-os-release\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksj8b\" (UniqueName: \"kubernetes.io/projected/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-kube-api-access-ksj8b\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-cni-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysctl-conf\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-tmp\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-cni-bin\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-cni-binary-copy\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-k8s-cni-cncf-io\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-kubelet\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysconfig\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysctl-d\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-tuned\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-systemd\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wnp\" (UniqueName: \"kubernetes.io/projected/31b8df2f-e374-4426-b777-905a07afac34-kube-api-access-t4wnp\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-netns\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.857063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e37d82-6241-46f1-974c-655e53ab2189-tmp-dir\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-socket-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2x2\" (UniqueName: \"kubernetes.io/projected/c3d94894-5060-45d1-bd7b-1fea2cedeb43-kube-api-access-9k2x2\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856901    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-cni-netd\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-daemon-config\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-host-slash\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.856991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eeead595-ea5f-4a2e-a731-bb8a56c7702a-agent-certs\") pod \"konnectivity-agent-dhr7p\" (UID: \"eeead595-ea5f-4a2e-a731-bb8a56c7702a\") " pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.857576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.857009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-multus-certs\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.957652 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-tmp\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-cni-bin\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-cni-binary-copy\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-k8s-cni-cncf-io\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-kubelet\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysconfig\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysctl-d\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-cni-bin\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.957841 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-kubelet\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-tuned\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysconfig\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957901    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysctl-d\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-k8s-cni-cncf-io\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-systemd\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wnp\" (UniqueName: \"kubernetes.io/projected/31b8df2f-e374-4426-b777-905a07afac34-kube-api-access-t4wnp\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.957991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-netns\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-systemd\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-netns\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e37d82-6241-46f1-974c-655e53ab2189-tmp-dir\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-socket-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2x2\" (UniqueName: \"kubernetes.io/projected/c3d94894-5060-45d1-bd7b-1fea2cedeb43-kube-api-access-9k2x2\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958142    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-cni-netd\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-daemon-config\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958234    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-host-slash\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.958259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eeead595-ea5f-4a2e-a731-bb8a56c7702a-agent-certs\") pod \"konnectivity-agent-dhr7p\" (UID: \"eeead595-ea5f-4a2e-a731-bb8a56c7702a\") " pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-multus-certs\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-host-slash\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-kubernetes\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958434    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-cni-netd\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-sys\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-cni-binary-copy\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt99\" (UniqueName: \"kubernetes.io/projected/25e37d82-6241-46f1-974c-655e53ab2189-kube-api-access-jrt99\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-sys\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-run-multus-certs\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-kubernetes\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-socket-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.958845    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.958938    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:44.458899424 +0000 UTC m=+4.880966270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:43.959230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e37d82-6241-46f1-974c-655e53ab2189-tmp-dir\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.958511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzhm\" (UniqueName: \"kubernetes.io/projected/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-kube-api-access-bfzhm\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-ovn\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-system-cni-dir\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-run-ovn\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-system-cni-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-cnibin\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-serviceca\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-systemd\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-cnibin\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eeead595-ea5f-4a2e-a731-bb8a56c7702a-konnectivity-ca\") pod \"konnectivity-agent-dhr7p\" (UID: \"eeead595-ea5f-4a2e-a731-bb8a56c7702a\") " pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-os-release\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-hostroot\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-system-cni-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h2t\" (UniqueName: \"kubernetes.io/projected/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-kube-api-access-d2h2t\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.959972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-systemd\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-daemon-config\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-device-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-system-cni-dir\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-hostroot\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-device-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959691    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-os-release\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-dbus\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-host\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-registration-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-host\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.960821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-run-netns\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-cni-bin\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-registration-dir\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-conf-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cnibin\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-slash\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-dbus\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-var-lib-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.959961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-var-lib-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960018    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-etc-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-log-socket\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-ovnkube-config\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960094    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-host\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-systemd-units\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-etc-kubernetes\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960206    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eeead595-ea5f-4a2e-a731-bb8a56c7702a-konnectivity-ca\") pod \"konnectivity-agent-dhr7p\" (UID: \"eeead595-ea5f-4a2e-a731-bb8a56c7702a\") " pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-serviceca\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-var-lib-kubelet\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.961656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-conf-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-etc-kubernetes\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-run-netns\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960481    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-systemd-units\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-cni-bin\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-log-socket\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.960915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-etc-openvswitch\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31b8df2f-e374-4426-b777-905a07afac34-ovn-node-metrics-cert\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-iptables-alerter-script\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-run\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-run-ovn-kubernetes\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-var-lib-kubelet\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-slash\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-cni-multus\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.962517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkjb\" (UniqueName: \"kubernetes.io/projected/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-kube-api-access-slkjb\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-cnibin\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-run-ovn-kubernetes\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.961951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-run\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-modprobe-d\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-modprobe-d\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-lib-modules\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-iptables-alerter-script\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-tmp\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-host-var-lib-cni-multus\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-host\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25e37d82-6241-46f1-974c-655e53ab2189-hosts-file\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.962976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-tuned\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-sys-fs\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963221    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eeead595-ea5f-4a2e-a731-bb8a56c7702a-agent-certs\") pod \"konnectivity-agent-dhr7p\" (UID: \"eeead595-ea5f-4a2e-a731-bb8a56c7702a\") " pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:43.963332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25e37d82-6241-46f1-974c-655e53ab2189-hosts-file\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-node-log\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-env-overrides\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-node-log\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8rq\" (UniqueName: \"kubernetes.io/projected/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-kube-api-access-xb8rq\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.963412    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-lib-modules\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-kubelet\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.963485    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:44.463460711 +0000 UTC m=+4.885527530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31b8df2f-e374-4426-b777-905a07afac34-host-kubelet\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-ovnkube-script-lib\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-sys-fs\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-socket-dir-parent\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874nk\" (UniqueName: \"kubernetes.io/projected/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-kube-api-access-874nk\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-kubelet-config\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-os-release\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.964280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.963989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksj8b\" (UniqueName: \"kubernetes.io/projected/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-kube-api-access-ksj8b\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-env-overrides\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-cni-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964086    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-kubelet-config\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysctl-conf\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964208    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-etc-sysctl-conf\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964272    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-cni-dir\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-os-release\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-multus-socket-dir-parent\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-ovnkube-script-lib\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.965037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.964953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31b8df2f-e374-4426-b777-905a07afac34-ovnkube-config\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.968436 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.968410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31b8df2f-e374-4426-b777-905a07afac34-ovn-node-metrics-cert\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.968579 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.968557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wnp\" (UniqueName: \"kubernetes.io/projected/31b8df2f-e374-4426-b777-905a07afac34-kube-api-access-t4wnp\") pod \"ovnkube-node-85shc\" (UID: \"31b8df2f-e374-4426-b777-905a07afac34\") " pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:43.968885 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.968858    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:29:43.968992 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.968891    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:29:43.968992 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.968906    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:43.968992 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:43.968964    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:29:44.468946148 +0000 UTC m=+4.891012984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:43.971502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.971475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2x2\" (UniqueName: \"kubernetes.io/projected/c3d94894-5060-45d1-bd7b-1fea2cedeb43-kube-api-access-9k2x2\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:43.971627 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.971574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt99\" (UniqueName: \"kubernetes.io/projected/25e37d82-6241-46f1-974c-655e53ab2189-kube-api-access-jrt99\") pod \"node-resolver-r8dkq\" (UID: \"25e37d82-6241-46f1-974c-655e53ab2189\") " pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:43.971627 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.971605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h2t\" (UniqueName: \"kubernetes.io/projected/c132a9e1-eb6e-4602-bfa2-40af3a44a8d6-kube-api-access-d2h2t\") pod \"node-ca-4dtpj\" (UID: \"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6\") " pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:43.972502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.972475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzhm\" (UniqueName: \"kubernetes.io/projected/fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a-kube-api-access-bfzhm\") pod \"aws-ebs-csi-driver-node-vjzpl\" (UID: \"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:43.972839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.972819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-874nk\" (UniqueName: \"kubernetes.io/projected/320d92f3-2f1e-45ec-8a37-fd0b31c5508f-kube-api-access-874nk\") pod \"multus-vlpg6\" (UID: \"320d92f3-2f1e-45ec-8a37-fd0b31c5508f\") " pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:43.973215 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.973196    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8rq\" (UniqueName: \"kubernetes.io/projected/94a1e6fa-04b0-465d-9ab7-d9f2abbc9890-kube-api-access-xb8rq\") pod \"tuned-fjqnz\" (UID: \"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890\") " pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:43.973797 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.973775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksj8b\" (UniqueName: \"kubernetes.io/projected/b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5-kube-api-access-ksj8b\") pod \"multus-additional-cni-plugins-tj678\" (UID: \"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5\") " pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:43.974346 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:43.974328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkjb\" (UniqueName: \"kubernetes.io/projected/bce92fb7-b0cb-4e49-8bf6-37e8c02f0133-kube-api-access-slkjb\") pod \"iptables-alerter-fdjrl\" (UID: \"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133\") " pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:44.084492 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.084442    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r8dkq"
Jul 24 01:29:44.107486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.107458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdjrl"
Jul 24 01:29:44.120999 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.120964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dtpj"
Jul 24 01:29:44.134693 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.134661    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tj678"
Jul 24 01:29:44.150398 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.150360    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fjqnz"
Jul 24 01:29:44.162961 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.162939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl"
Jul 24 01:29:44.179725 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.179695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:29:44.186482 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.186292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:29:44.190981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.190962    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlpg6"
Jul 24 01:29:44.354605 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.354571    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496a950db9b46b88536b00684a81f3f6.slice/crio-0f73f902eaae9ffc9cae77cc17a0fa5b8aa1de972d8cd8d69fc9f117f562e125 WatchSource:0}: Error finding container 0f73f902eaae9ffc9cae77cc17a0fa5b8aa1de972d8cd8d69fc9f117f562e125: Status 404 returned error can't find the container with id 0f73f902eaae9ffc9cae77cc17a0fa5b8aa1de972d8cd8d69fc9f117f562e125
Jul 24 01:29:44.362603 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.362585    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 01:29:44.368932 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.368910    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31744d3a57961580d2625e5c1e1dcc8b.slice/crio-37e0d440b0a4706772ed2650177138b5dc5ad46e4900b98ed1a6a8bf98fb2a24 WatchSource:0}: Error finding container 37e0d440b0a4706772ed2650177138b5dc5ad46e4900b98ed1a6a8bf98fb2a24: Status 404 returned error can't find the container with id 37e0d440b0a4706772ed2650177138b5dc5ad46e4900b98ed1a6a8bf98fb2a24
Jul 24 01:29:44.441703 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.441671    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e37d82_6241_46f1_974c_655e53ab2189.slice/crio-a097b8d02ca28b83c9c817d3f6655b25aaf94729f942318d32b831883b4093f1 WatchSource:0}: Error finding container a097b8d02ca28b83c9c817d3f6655b25aaf94729f942318d32b831883b4093f1: Status 404 returned error can't find the container with id a097b8d02ca28b83c9c817d3f6655b25aaf94729f942318d32b831883b4093f1
Jul 24 01:29:44.467724 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.467688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:44.467867 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.467746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:44.467867 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.467862    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:44.467942 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.467916    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:45.467901356 +0000 UTC m=+5.889968172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:44.468001 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.467863    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:44.468037 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.468012    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:45.467992611 +0000 UTC m=+5.890059445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:44.520426 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.520389    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc132a9e1_eb6e_4602_bfa2_40af3a44a8d6.slice/crio-002bbc7cc7d3cbcba8bb92846e9b33a97fef59bbc99fda3b0ff2ffe7d24bbce3 WatchSource:0}: Error finding container 002bbc7cc7d3cbcba8bb92846e9b33a97fef59bbc99fda3b0ff2ffe7d24bbce3: Status 404 returned error can't find the container with id 002bbc7cc7d3cbcba8bb92846e9b33a97fef59bbc99fda3b0ff2ffe7d24bbce3
Jul 24 01:29:44.520997 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.520974    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b8df2f_e374_4426_b777_905a07afac34.slice/crio-3fcc257bbce20902d25ec7396c025db4218d58037ed957129f5083d61671fc88 WatchSource:0}: Error finding container 3fcc257bbce20902d25ec7396c025db4218d58037ed957129f5083d61671fc88: Status 404 returned error can't find the container with id 3fcc257bbce20902d25ec7396c025db4218d58037ed957129f5083d61671fc88
Jul 24 01:29:44.521867 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.521826    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce92fb7_b0cb_4e49_8bf6_37e8c02f0133.slice/crio-05c64a3338ff6a5e746d2a608321a2e47ff1c568942b94a24eab4a2b06d9cdbd WatchSource:0}: Error finding container 05c64a3338ff6a5e746d2a608321a2e47ff1c568942b94a24eab4a2b06d9cdbd: Status 404 returned error can't find the container with id 05c64a3338ff6a5e746d2a608321a2e47ff1c568942b94a24eab4a2b06d9cdbd
Jul 24 01:29:44.541378 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.541354    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320d92f3_2f1e_45ec_8a37_fd0b31c5508f.slice/crio-d6d2810b47996365250e562c3c13e989d8ded27f1a8ac91dbaf44f8904cbedb6 WatchSource:0}: Error finding container d6d2810b47996365250e562c3c13e989d8ded27f1a8ac91dbaf44f8904cbedb6: Status 404 returned error can't find the container with id d6d2810b47996365250e562c3c13e989d8ded27f1a8ac91dbaf44f8904cbedb6
Jul 24 01:29:44.561237 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.561211    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a1e6fa_04b0_465d_9ab7_d9f2abbc9890.slice/crio-f6ea4a129085b33c30a6058aa00734cfb27e57a63136e4d710181eb1e9dabdf4 WatchSource:0}: Error finding container f6ea4a129085b33c30a6058aa00734cfb27e57a63136e4d710181eb1e9dabdf4: Status 404 returned error can't find the container with id f6ea4a129085b33c30a6058aa00734cfb27e57a63136e4d710181eb1e9dabdf4
Jul 24 01:29:44.568443 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.568419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:44.568587 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.568563    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:29:44.568587 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.568581    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:29:44.568687 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.568591    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:44.568687 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:44.568645    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:29:45.568622019 +0000 UTC m=+5.990688834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:44.588473 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.588446    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f0c07f_1466_4cdb_9f6c_bdeb36e089f5.slice/crio-fde025fcddc665e21fc62f4b5488ad73e6c3427910267f894004b09106e1bec9 WatchSource:0}: Error finding container fde025fcddc665e21fc62f4b5488ad73e6c3427910267f894004b09106e1bec9: Status 404 returned error can't find the container with id fde025fcddc665e21fc62f4b5488ad73e6c3427910267f894004b09106e1bec9
Jul 24 01:29:44.632064 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.632035    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb85c0bd_b49b_44ad_9f41_b3fb4ebdc51a.slice/crio-d14a02a3516ee020f056deef2afcb63bf9ca124fd2797e5c525e08f3a3c71feb WatchSource:0}: Error finding container d14a02a3516ee020f056deef2afcb63bf9ca124fd2797e5c525e08f3a3c71feb: Status 404 returned error can't find the container with id d14a02a3516ee020f056deef2afcb63bf9ca124fd2797e5c525e08f3a3c71feb
Jul 24 01:29:44.639158 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:29:44.639126    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeead595_ea5f_4a2e_a731_bb8a56c7702a.slice/crio-58a3eca83c31aae7bb5b2e8879b00effdc689541b6e837d6fb9656fb314f6fb3 WatchSource:0}: Error finding container 58a3eca83c31aae7bb5b2e8879b00effdc689541b6e837d6fb9656fb314f6fb3: Status 404 returned error can't find the container with id 58a3eca83c31aae7bb5b2e8879b00effdc689541b6e837d6fb9656fb314f6fb3
Jul 24 01:29:44.744968 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.744905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"3fcc257bbce20902d25ec7396c025db4218d58037ed957129f5083d61671fc88"}
Jul 24 01:29:44.746018 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.745995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dtpj" event={"ID":"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6","Type":"ContainerStarted","Data":"002bbc7cc7d3cbcba8bb92846e9b33a97fef59bbc99fda3b0ff2ffe7d24bbce3"}
Jul 24 01:29:44.747118 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.747095    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal" event={"ID":"496a950db9b46b88536b00684a81f3f6","Type":"ContainerStarted","Data":"0f73f902eaae9ffc9cae77cc17a0fa5b8aa1de972d8cd8d69fc9f117f562e125"}
Jul 24 01:29:44.748083 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.748061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dhr7p" event={"ID":"eeead595-ea5f-4a2e-a731-bb8a56c7702a","Type":"ContainerStarted","Data":"58a3eca83c31aae7bb5b2e8879b00effdc689541b6e837d6fb9656fb314f6fb3"}
Jul 24 01:29:44.749118 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.749092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerStarted","Data":"fde025fcddc665e21fc62f4b5488ad73e6c3427910267f894004b09106e1bec9"}
Jul 24 01:29:44.750233 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.750214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r8dkq" event={"ID":"25e37d82-6241-46f1-974c-655e53ab2189","Type":"ContainerStarted","Data":"a097b8d02ca28b83c9c817d3f6655b25aaf94729f942318d32b831883b4093f1"}
Jul 24 01:29:44.751261 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.751232    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal" event={"ID":"31744d3a57961580d2625e5c1e1dcc8b","Type":"ContainerStarted","Data":"37e0d440b0a4706772ed2650177138b5dc5ad46e4900b98ed1a6a8bf98fb2a24"}
Jul 24 01:29:44.752311 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.752290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl" event={"ID":"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a","Type":"ContainerStarted","Data":"d14a02a3516ee020f056deef2afcb63bf9ca124fd2797e5c525e08f3a3c71feb"}
Jul 24 01:29:44.753398 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.753368    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fjqnz" event={"ID":"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890","Type":"ContainerStarted","Data":"f6ea4a129085b33c30a6058aa00734cfb27e57a63136e4d710181eb1e9dabdf4"}
Jul 24 01:29:44.754435 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.754417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlpg6" event={"ID":"320d92f3-2f1e-45ec-8a37-fd0b31c5508f","Type":"ContainerStarted","Data":"d6d2810b47996365250e562c3c13e989d8ded27f1a8ac91dbaf44f8904cbedb6"}
Jul 24 01:29:44.755445 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.755428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdjrl" event={"ID":"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133","Type":"ContainerStarted","Data":"05c64a3338ff6a5e746d2a608321a2e47ff1c568942b94a24eab4a2b06d9cdbd"}
Jul 24 01:29:44.792541 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.792490    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 01:24:44 +0000 UTC" deadline="2028-01-10 11:46:20.724902176 +0000 UTC"
Jul 24 01:29:44.792541 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:44.792534    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12850h16m35.932371406s"
Jul 24 01:29:45.203558 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.203520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:45.203758 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.203705    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:45.203828 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.203818    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:45.203956 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.203934    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:45.204032 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.204014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:45.204147 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.204118    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:45.476175 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.475251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:45.476175 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.475347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:45.476175 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.475551    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:45.476175 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.475621    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:47.475601227 +0000 UTC m=+7.897668043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:45.476175 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.476063    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:45.476175 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.476131    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:47.476100011 +0000 UTC m=+7.898166846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:45.576796 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.576752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:45.577024 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.576938    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:29:45.577024 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.576962    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:29:45.577024 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.576975    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:45.577259 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:45.577095    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:29:47.577053154 +0000 UTC m=+7.999119971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:45.793654 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.793487    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 01:24:44 +0000 UTC" deadline="2027-12-29 18:06:45.047551335 +0000 UTC"
Jul 24 01:29:45.793654 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:45.793539    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12568h36m59.254017489s"
Jul 24 01:29:47.204360 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:47.204318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:47.204924 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.204469    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:47.204924 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:47.204570    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:47.204924 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.204664    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:47.204924 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:47.204734    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:47.204924 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.204805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:47.500756 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:47.500663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:47.500937 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:47.500736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:47.500997 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.500949    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:47.501064 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.501022    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:51.501001815 +0000 UTC m=+11.923068633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:47.501776 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.501475    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:47.501776 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.501535    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:51.501518424 +0000 UTC m=+11.923585243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:47.603243 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:47.603202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:47.603451 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.603424    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:29:47.603451 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.603446    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:29:47.603564 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.603460    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:47.603564 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:47.603525    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:29:51.603505014 +0000 UTC m=+12.025571834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:49.203837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:49.203803    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:49.203837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:49.203833    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:49.204466 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:49.204003    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:49.204466 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:49.204015    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:49.204466 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:49.204143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:49.204466 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:49.204247    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:51.204177 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:51.204136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:51.204636 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:51.204144    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:51.204636 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.204279    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:51.204636 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.204392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:51.204636 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:51.204157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:51.204636 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.204473    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:51.537997 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:51.537301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:51.537997 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:51.537394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:51.537997 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.537532    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:51.537997 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.537595    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:59.537576301 +0000 UTC m=+19.959643122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:51.537997 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.537824    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:51.537997 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.537914    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:29:59.537864998 +0000 UTC m=+19.959931821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:51.638517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:51.638417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:51.638517 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.638471    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:29:51.638517 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.638500    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:29:51.638517 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.638513    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:51.638834 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:51.638573    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:29:59.638556291 +0000 UTC m=+20.060623106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:53.204407 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:53.204347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:53.204407 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:53.204397    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:53.204951 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:53.204511    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:53.204951 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:53.204840    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:53.204951 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:53.204931    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:53.205186 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:53.205039    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:55.203780 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:55.203742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:55.204258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:55.203742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:55.204258 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:55.203868    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:55.204258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:55.203743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:55.204258 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:55.203921    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:55.204258 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:55.204044    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:57.203808 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:57.203768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:57.203808 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:57.203800    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:57.204379 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:57.203769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:57.204379 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:57.203909    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:57.204379 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:57.203995    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:57.204379 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:57.204074    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:59.203349 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:59.203316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:59.203793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:59.203316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:59.203793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.203452    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:29:59.203793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:59.203316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:59.203793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.203518    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:29:59.203793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.203583    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:29:59.601246 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:59.601162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:29:59.601246 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:59.601225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:29:59.601463 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.601332    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:59.601463 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.601334    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:59.601463 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.601396    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:15.601376175 +0000 UTC m=+36.023443013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:29:59.601463 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.601413    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:15.601405935 +0000 UTC m=+36.023472748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:29:59.702105 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:29:59.702075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:29:59.702275 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.702205    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:29:59.702275 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.702231    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:29:59.702275 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.702245    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:29:59.702465 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:29:59.702315    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:30:15.702298171 +0000 UTC m=+36.124364987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:30:01.204064 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:01.203997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:01.204064 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:01.204063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:01.204579 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:01.204014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:01.204579 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:01.204160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:01.204579 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:01.204238    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:01.204579 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:01.204290    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:03.203668 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:03.203627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:03.203668 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:03.203648    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:03.204159 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:03.203748    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:03.204159 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:03.203796    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:03.204159 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:03.203939    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:03.204159 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:03.204020    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:04.816974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.816651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal" event={"ID":"496a950db9b46b88536b00684a81f3f6","Type":"ContainerStarted","Data":"7534091b5eb558f2e657072dd3a77249d4b06ee36b6acb1d2bd21f59c62d302c"}
Jul 24 01:30:04.818299 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.818276    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fjqnz" event={"ID":"94a1e6fa-04b0-465d-9ab7-d9f2abbc9890","Type":"ContainerStarted","Data":"892440a4699869c3de94e0f6ab8b26b496289c00be7a4c68bbed2f7f062f708e"}
Jul 24 01:30:04.820671 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.820641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlpg6" event={"ID":"320d92f3-2f1e-45ec-8a37-fd0b31c5508f","Type":"ContainerStarted","Data":"ad245c9d7c6956598e5b0baf2f05ae165ebb5d17fb15c750df60108bc63b5881"}
Jul 24 01:30:04.823802 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.823785    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:30:04.824208 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.824187    2564 generic.go:356] "Generic (PLEG): container finished" podID="31b8df2f-e374-4426-b777-905a07afac34" containerID="cfcff54fb30f7784841d598df6c612125aa1fe371f69af0ecf35c63ccd8e9311" exitCode=1
Jul 24 01:30:04.824285 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.824221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"9dcdf96af16049c8bfeee49bdd7e563e4c64967674a68f41dcd86723846918f6"}
Jul 24 01:30:04.824285 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.824275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"e4dcfc4767a76b759053c70778cecfa068f2b106a0c1885f34731a83a363523b"}
Jul 24 01:30:04.824355 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.824314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"ff4617fcf21422948b1c49978ea6a147c73c165e219a3d83f196d2ea5cb3c9d9"}
Jul 24 01:30:04.824355 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.824343    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerDied","Data":"cfcff54fb30f7784841d598df6c612125aa1fe371f69af0ecf35c63ccd8e9311"}
Jul 24 01:30:04.824416 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.824378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"a615184d91af36c52cba9635af413d0f66c663be1f14ae3d7f2bc989f2489bcc"}
Jul 24 01:30:04.829618 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.829578    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-127.ec2.internal" podStartSLOduration=21.82956328 podStartE2EDuration="21.82956328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:04.829421618 +0000 UTC m=+25.251488468" watchObservedRunningTime="2026-07-24 01:30:04.82956328 +0000 UTC m=+25.251630115"
Jul 24 01:30:04.844544 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.844491    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fjqnz" podStartSLOduration=2.499179396 podStartE2EDuration="21.844471943s" totalImagesPullingTime="19.345292547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:04.843583584 +0000 UTC m=+25.265650420" watchObservedRunningTime="2026-07-24 01:30:04.844471943 +0000 UTC m=+25.266538780"
Jul 24 01:30:04.860460 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:04.860395    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vlpg6" podStartSLOduration=2.458344461 podStartE2EDuration="21.8603775s" totalImagesPullingTime="19.402033039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:04.86036677 +0000 UTC m=+25.282433607" watchObservedRunningTime="2026-07-24 01:30:04.8603775 +0000 UTC m=+25.282444337"
Jul 24 01:30:05.204200 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.204113    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:05.204200 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.204157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:05.204383 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.204225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:05.204383 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:05.204240    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:05.204383 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:05.204304    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:05.204383 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:05.204369    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:05.828733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.828697    2564 generic.go:356] "Generic (PLEG): container finished" podID="31744d3a57961580d2625e5c1e1dcc8b" containerID="e5e39aeed23016b2da34d8cd2d1612dfe5576048ed2eb22087aeec90be01914d" exitCode=0
Jul 24 01:30:05.829770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.828742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal" event={"ID":"31744d3a57961580d2625e5c1e1dcc8b","Type":"ContainerDied","Data":"e5e39aeed23016b2da34d8cd2d1612dfe5576048ed2eb22087aeec90be01914d"}
Jul 24 01:30:05.830722 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.830692    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl" event={"ID":"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a","Type":"ContainerStarted","Data":"7e635d121964f2bbfcbcc48ad33f46b64e9e19565acd7e83d75c077c0daf22e4"}
Jul 24 01:30:05.832433 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.832407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdjrl" event={"ID":"bce92fb7-b0cb-4e49-8bf6-37e8c02f0133","Type":"ContainerStarted","Data":"636e72e98482704c98d59b27d5f266179e23a12e1834638c0524df7a1d66f09e"}
Jul 24 01:30:05.835423 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.835397    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:30:05.835820 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.835797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"485fdcaf4f204a1ea65c3d6809a602cf9de2edfb0c1894705c7c19d7a0523575"}
Jul 24 01:30:05.837332 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.837314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dtpj" event={"ID":"c132a9e1-eb6e-4602-bfa2-40af3a44a8d6","Type":"ContainerStarted","Data":"10cd0e237294f1a646e966b374cb9e7805f7468c57f549795dee83cc270f2c4d"}
Jul 24 01:30:05.838815 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.838768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dhr7p" event={"ID":"eeead595-ea5f-4a2e-a731-bb8a56c7702a","Type":"ContainerStarted","Data":"62ef4a2b070234ece0ec155ca542f0c3dd60595aed7e3b1f5c127c59b3155eec"}
Jul 24 01:30:05.840360 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.840337    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5" containerID="22dd424b61b0d72c2be463046703556672d08b74e5b107f6a61de07443507a05" exitCode=0
Jul 24 01:30:05.840479 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.840417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerDied","Data":"22dd424b61b0d72c2be463046703556672d08b74e5b107f6a61de07443507a05"}
Jul 24 01:30:05.841991 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.841971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r8dkq" event={"ID":"25e37d82-6241-46f1-974c-655e53ab2189","Type":"ContainerStarted","Data":"141ca93ba185ad29a673366d92a1030cbe4d811ce23102361050cacf4315e05b"}
Jul 24 01:30:05.855118 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.855062    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fdjrl" podStartSLOduration=12.170870038 podStartE2EDuration="23.855020941s" totalImagesPullingTime="11.684150903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:05.854367307 +0000 UTC m=+26.276434161" watchObservedRunningTime="2026-07-24 01:30:05.855020941 +0000 UTC m=+26.277087778"
Jul 24 01:30:05.901220 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.901010    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dhr7p" podStartSLOduration=3.6779959 podStartE2EDuration="22.900995493s" totalImagesPullingTime="19.222999593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:05.887481681 +0000 UTC m=+26.309548517" watchObservedRunningTime="2026-07-24 01:30:05.900995493 +0000 UTC m=+26.323062329"
Jul 24 01:30:05.901680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.901655    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4dtpj" podStartSLOduration=4.566910665 podStartE2EDuration="23.901642012s" totalImagesPullingTime="19.334731347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:05.900809183 +0000 UTC m=+26.322876029" watchObservedRunningTime="2026-07-24 01:30:05.901642012 +0000 UTC m=+26.323708848"
Jul 24 01:30:05.922646 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:05.922590    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r8dkq" podStartSLOduration=12.15344031 podStartE2EDuration="23.922576814s" totalImagesPullingTime="11.769136504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:05.922207215 +0000 UTC m=+26.344274052" watchObservedRunningTime="2026-07-24 01:30:05.922576814 +0000 UTC m=+26.344643649"
Jul 24 01:30:06.540634 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:06.540601    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 01:30:06.845842 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:06.845807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal" event={"ID":"31744d3a57961580d2625e5c1e1dcc8b","Type":"ContainerStarted","Data":"3461c95c09d6c4d078a1070c4325fd81931550de654bd1b52844c5831f9e029a"}
Jul 24 01:30:06.847601 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:06.847575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl" event={"ID":"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a","Type":"ContainerStarted","Data":"071ee9bec3b6abdb01d5f10c6358494c129699948639314e7224219beddd13f4"}
Jul 24 01:30:06.859528 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:06.859479    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-127.ec2.internal" podStartSLOduration=23.859463057 podStartE2EDuration="23.859463057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:06.859255833 +0000 UTC m=+27.281322690" watchObservedRunningTime="2026-07-24 01:30:06.859463057 +0000 UTC m=+27.281529895"
Jul 24 01:30:07.076683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.076570    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T01:30:06.540624123Z","UUID":"04b24ab1-fdc7-459d-8b64-b682ab475cf9","Handler":null,"Name":"","Endpoint":""}
Jul 24 01:30:07.078447 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.078427    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 01:30:07.078570 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.078471    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 01:30:07.203715 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.203682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:07.203934 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.203682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:07.203934 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:07.203816    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:07.203934 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.203699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:07.203934 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:07.203882    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:07.204174 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:07.203934    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:07.854353 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.854299    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:30:07.857060 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.856552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"57fcd4b664a07a2748d012dacc39c0be18c6acd03b09d867a8d154e52ab4d2d9"}
Jul 24 01:30:07.860065 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.860018    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl" event={"ID":"fb85c0bd-b49b-44ad-9f41-b3fb4ebdc51a","Type":"ContainerStarted","Data":"6bccdb97fb87f0646e0d2ee1d56f61fbd85bc83dcf98123328c81aab7b5792e9"}
Jul 24 01:30:07.877633 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:07.877588    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjzpl" podStartSLOduration=3.072731153 podStartE2EDuration="24.877573574s" totalImagesPullingTime="21.804842421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:07.877090122 +0000 UTC m=+28.299156978" watchObservedRunningTime="2026-07-24 01:30:07.877573574 +0000 UTC m=+28.299640410"
Jul 24 01:30:08.538339 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:08.538287    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:30:08.539036 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:08.539002    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:30:08.862853 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:08.862763    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:30:08.863369 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:08.863271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dhr7p"
Jul 24 01:30:09.203950 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:09.203885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:09.204148 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:09.203905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:09.204148 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:09.204043    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:09.204148 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:09.204055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:09.204309 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:09.204153    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:09.204309 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:09.204282    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:10.870919 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.870041    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5" containerID="6f6bd79c3444fdb4b1dd39afcd9aa03f4cb0a979fd8dd9cf9cb49f2e0aad051d" exitCode=0
Jul 24 01:30:10.870919 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.870769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerDied","Data":"6f6bd79c3444fdb4b1dd39afcd9aa03f4cb0a979fd8dd9cf9cb49f2e0aad051d"}
Jul 24 01:30:10.876616 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.876594    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:30:10.876942 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.876920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"84cab3830adc163b67f7bdc0600213ac052c0ad6ca1f0ce7d5b3a483a5cc9126"}
Jul 24 01:30:10.877214 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.877197    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:30:10.877269 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.877261    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:30:10.877325 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.877309    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:30:10.877383 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.877364    2564 scope.go:117] "RemoveContainer" containerID="cfcff54fb30f7784841d598df6c612125aa1fe371f69af0ecf35c63ccd8e9311"
Jul 24 01:30:10.893596 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.893578    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:30:10.893738 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:10.893723    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:30:11.203505 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:11.203474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:11.203646 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:11.203474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:11.203646 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:11.203570    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:11.203792 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:11.203675    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:11.203792 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:11.203474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:11.203792 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:11.203746    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:11.887099 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:11.886865    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:30:11.887506 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:11.887268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" event={"ID":"31b8df2f-e374-4426-b777-905a07afac34","Type":"ContainerStarted","Data":"961c72d1d94b1f59f081461eb8733dc9ef18b68d0214d882b843f5577941c38b"}
Jul 24 01:30:11.920978 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:11.919956    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-85shc" podStartSLOduration=9.225374169 podStartE2EDuration="28.919926624s" totalImagesPullingTime="19.694552455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:11.916051037 +0000 UTC m=+32.338117895" watchObservedRunningTime="2026-07-24 01:30:11.919926624 +0000 UTC m=+32.341993463"
Jul 24 01:30:12.591501 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.591234    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2dzp8"]
Jul 24 01:30:12.591647 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.591629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:12.591780 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:12.591754    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:12.594277 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.594249    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9kkdx"]
Jul 24 01:30:12.594443 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.594352    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:12.594506 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:12.594443    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:12.594848 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.594827    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crts2"]
Jul 24 01:30:12.594943 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.594930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:12.595072 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:12.595047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:12.891693 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.891608    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5" containerID="57fed89ba9fcb05646a7a003217632c6102b9a623025acf32c3cf3729301b6ab" exitCode=0
Jul 24 01:30:12.891693 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:12.891663    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerDied","Data":"57fed89ba9fcb05646a7a003217632c6102b9a623025acf32c3cf3729301b6ab"}
Jul 24 01:30:13.895890 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:13.895807    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5" containerID="fa19c5544fba0c60fcfe27a13f5fd1e01c246de88308250fe4ef58da702b4960" exitCode=0
Jul 24 01:30:13.895890 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:13.895882    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerDied","Data":"fa19c5544fba0c60fcfe27a13f5fd1e01c246de88308250fe4ef58da702b4960"}
Jul 24 01:30:14.204187 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:14.204150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:14.204346 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:14.204150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:14.204346 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:14.204271    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:14.204346 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:14.204280    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:14.204476 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:14.204354    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:14.204476 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:14.204422    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:15.617669 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:15.617581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:15.617669 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:15.617639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:15.618181 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.617746    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:30:15.618181 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.617746    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 01:30:15.618181 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.617844    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs podName:c3d94894-5060-45d1-bd7b-1fea2cedeb43 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:47.617797015 +0000 UTC m=+68.039863833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs") pod "network-metrics-daemon-crts2" (UID: "c3d94894-5060-45d1-bd7b-1fea2cedeb43") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 01:30:15.618181 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.617872    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret podName:ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:47.617860186 +0000 UTC m=+68.039927000 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret") pod "global-pull-secret-syncer-2dzp8" (UID: "ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53") : object "kube-system"/"original-pull-secret" not registered
Jul 24 01:30:15.718402 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:15.718350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:15.718573 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.718526    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 01:30:15.718573 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.718547    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 01:30:15.718573 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.718562    2564 projected.go:196] Error preparing data for projected volume kube-api-access-7db9v for pod openshift-network-diagnostics/network-check-target-9kkdx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:30:15.718687 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:15.718612    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v podName:4934861c-f221-4082-b600-b6f14c26185b nodeName:}" failed. No retries permitted until 2026-07-24 01:30:47.718598897 +0000 UTC m=+68.140665711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7db9v" (UniqueName: "kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v") pod "network-check-target-9kkdx" (UID: "4934861c-f221-4082-b600-b6f14c26185b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 01:30:16.203744 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:16.203706    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:16.203744 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:16.203748    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:16.203962 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:16.203714    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:16.203962 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:16.203836    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dzp8" podUID="ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53"
Jul 24 01:30:16.203962 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:16.203935    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crts2" podUID="c3d94894-5060-45d1-bd7b-1fea2cedeb43"
Jul 24 01:30:16.204147 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:16.204020    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kkdx" podUID="4934861c-f221-4082-b600-b6f14c26185b"
Jul 24 01:30:17.382624 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.382534    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-127.ec2.internal" event="NodeReady"
Jul 24 01:30:17.383125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.382696    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 01:30:17.417078 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.417047    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8577649d7d-4btcd"]
Jul 24 01:30:17.422476 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.422446    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.425159 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.425135    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 01:30:17.425294 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.425138    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ld5t9\""
Jul 24 01:30:17.425539 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.425514    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 01:30:17.425669 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.425637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 01:30:17.430956 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.430920    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 01:30:17.433538 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.433508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8577649d7d-4btcd"]
Jul 24 01:30:17.434375 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.434354    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z6ckz"]
Jul 24 01:30:17.437928 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.437908    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qj8mc"]
Jul 24 01:30:17.438060 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.438043    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.440609 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.440586    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8g76\""
Jul 24 01:30:17.440609 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.440602    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 01:30:17.440770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.440634    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 01:30:17.441240 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.441222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.443772 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.443641    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 01:30:17.443772 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.443655    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tfscd\""
Jul 24 01:30:17.443935 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.443827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 01:30:17.443994 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.443985    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 01:30:17.447828 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.447807    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qj8mc"]
Jul 24 01:30:17.448746 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.448694    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z6ckz"]
Jul 24 01:30:17.531256 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-installation-pull-secrets\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531256 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-kube-api-access-nzk7l\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae8f6899-df6c-445c-98b4-856f9af10e14-ca-trust-extracted\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-certificates\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d92p\" (UniqueName: \"kubernetes.io/projected/432593c2-05d4-4dfc-8246-87cc5b5d91c5-kube-api-access-8d92p\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.531504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531480    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-bound-sa-token\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432593c2-05d4-4dfc-8246-87cc5b5d91c5-tmp-dir\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-image-registry-private-configuration\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797c9bee-eab6-41f0-a52a-e64cd1034f1f-cert\") pod \"ingress-canary-qj8mc\" (UID: \"797c9bee-eab6-41f0-a52a-e64cd1034f1f\") " pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/432593c2-05d4-4dfc-8246-87cc5b5d91c5-config-volume\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-trusted-ca\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.531755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.531681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw59m\" (UniqueName: \"kubernetes.io/projected/797c9bee-eab6-41f0-a52a-e64cd1034f1f-kube-api-access-bw59m\") pod \"ingress-canary-qj8mc\" (UID: \"797c9bee-eab6-41f0-a52a-e64cd1034f1f\") " pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.632600 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.632560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/432593c2-05d4-4dfc-8246-87cc5b5d91c5-config-volume\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.632600 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.632599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-trusted-ca\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.632878 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.632622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.632878 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.632650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw59m\" (UniqueName: \"kubernetes.io/projected/797c9bee-eab6-41f0-a52a-e64cd1034f1f-kube-api-access-bw59m\") pod \"ingress-canary-qj8mc\" (UID: \"797c9bee-eab6-41f0-a52a-e64cd1034f1f\") " pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.632878 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:17.632753    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 01:30:17.632878 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:17.632835    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls podName:432593c2-05d4-4dfc-8246-87cc5b5d91c5 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:18.132812895 +0000 UTC m=+38.554879708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls") pod "dns-default-z6ckz" (UID: "432593c2-05d4-4dfc-8246-87cc5b5d91c5") : secret "dns-default-metrics-tls" not found
Jul 24 01:30:17.633117 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.632985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-installation-pull-secrets\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633117 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-kube-api-access-nzk7l\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633117 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae8f6899-df6c-445c-98b4-856f9af10e14-ca-trust-extracted\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633117 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-certificates\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633363 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633130    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d92p\" (UniqueName: \"kubernetes.io/projected/432593c2-05d4-4dfc-8246-87cc5b5d91c5-kube-api-access-8d92p\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.633363 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/432593c2-05d4-4dfc-8246-87cc5b5d91c5-config-volume\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.633363 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633494 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-bound-sa-token\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633494 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432593c2-05d4-4dfc-8246-87cc5b5d91c5-tmp-dir\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.633494 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-image-registry-private-configuration\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633494 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797c9bee-eab6-41f0-a52a-e64cd1034f1f-cert\") pod \"ingress-canary-qj8mc\" (UID: \"797c9bee-eab6-41f0-a52a-e64cd1034f1f\") " pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.633616 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae8f6899-df6c-445c-98b4-856f9af10e14-ca-trust-extracted\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633691 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:17.633666    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 01:30:17.633691 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:17.633688    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8577649d7d-4btcd: secret "image-registry-tls" not found
Jul 24 01:30:17.633813 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633723    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-trusted-ca\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633813 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:17.633734    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls podName:ae8f6899-df6c-445c-98b4-856f9af10e14 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:18.133716956 +0000 UTC m=+38.555783785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls") pod "image-registry-8577649d7d-4btcd" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14") : secret "image-registry-tls" not found
Jul 24 01:30:17.633813 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-certificates\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.633970 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.633808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432593c2-05d4-4dfc-8246-87cc5b5d91c5-tmp-dir\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.638001 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.637854    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-image-registry-private-configuration\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.638146 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.637861    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-installation-pull-secrets\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.638146 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.637863    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797c9bee-eab6-41f0-a52a-e64cd1034f1f-cert\") pod \"ingress-canary-qj8mc\" (UID: \"797c9bee-eab6-41f0-a52a-e64cd1034f1f\") " pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.641965 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.641940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw59m\" (UniqueName: \"kubernetes.io/projected/797c9bee-eab6-41f0-a52a-e64cd1034f1f-kube-api-access-bw59m\") pod \"ingress-canary-qj8mc\" (UID: \"797c9bee-eab6-41f0-a52a-e64cd1034f1f\") " pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:17.642128 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.642104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-kube-api-access-nzk7l\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.642388 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.642365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-bound-sa-token\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:17.642551 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.642535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d92p\" (UniqueName: \"kubernetes.io/projected/432593c2-05d4-4dfc-8246-87cc5b5d91c5-kube-api-access-8d92p\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:17.755626 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:17.755584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qj8mc"
Jul 24 01:30:18.137345 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.137310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:18.137519 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.137384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:18.137519 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:18.137487    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 01:30:18.137519 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:18.137503    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 01:30:18.137519 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:18.137517    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8577649d7d-4btcd: secret "image-registry-tls" not found
Jul 24 01:30:18.137716 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:18.137570    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls podName:432593c2-05d4-4dfc-8246-87cc5b5d91c5 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:19.137548834 +0000 UTC m=+39.559615651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls") pod "dns-default-z6ckz" (UID: "432593c2-05d4-4dfc-8246-87cc5b5d91c5") : secret "dns-default-metrics-tls" not found
Jul 24 01:30:18.137716 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:18.137587    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls podName:ae8f6899-df6c-445c-98b4-856f9af10e14 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:19.137580573 +0000 UTC m=+39.559647386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls") pod "image-registry-8577649d7d-4btcd" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14") : secret "image-registry-tls" not found
Jul 24 01:30:18.204092 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.204059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:18.204279 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.204059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:18.204279 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.204072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:18.206741 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.206721    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 01:30:18.206866 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.206745    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 01:30:18.207873 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.207643    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c62r2\""
Jul 24 01:30:18.207873 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.207736    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 01:30:18.207873 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.207743    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pnzwr\""
Jul 24 01:30:18.207873 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.207759    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 01:30:18.401816 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.401784    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qj8mc"]
Jul 24 01:30:18.413006 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:18.412981    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797c9bee_eab6_41f0_a52a_e64cd1034f1f.slice/crio-23b042b58c8da00bda90ab383ecf110df4eec809392879edce9194162325f0b7 WatchSource:0}: Error finding container 23b042b58c8da00bda90ab383ecf110df4eec809392879edce9194162325f0b7: Status 404 returned error can't find the container with id 23b042b58c8da00bda90ab383ecf110df4eec809392879edce9194162325f0b7
Jul 24 01:30:18.910725 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.910645    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5" containerID="ff7ece2726939ee32d8c94c16fe48e08c714409c2874e6524849833ee8b28f15" exitCode=0
Jul 24 01:30:18.910899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.910735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerDied","Data":"ff7ece2726939ee32d8c94c16fe48e08c714409c2874e6524849833ee8b28f15"}
Jul 24 01:30:18.912093 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:18.912074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qj8mc" event={"ID":"797c9bee-eab6-41f0-a52a-e64cd1034f1f","Type":"ContainerStarted","Data":"23b042b58c8da00bda90ab383ecf110df4eec809392879edce9194162325f0b7"}
Jul 24 01:30:19.144464 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:19.144429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:19.144653 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:19.144510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:19.144653 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:19.144569    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 01:30:19.144653 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:19.144621    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 01:30:19.144653 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:19.144632    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls podName:432593c2-05d4-4dfc-8246-87cc5b5d91c5 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:21.144614421 +0000 UTC m=+41.566681235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls") pod "dns-default-z6ckz" (UID: "432593c2-05d4-4dfc-8246-87cc5b5d91c5") : secret "dns-default-metrics-tls" not found
Jul 24 01:30:19.144653 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:19.144635    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8577649d7d-4btcd: secret "image-registry-tls" not found
Jul 24 01:30:19.144847 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:19.144684    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls podName:ae8f6899-df6c-445c-98b4-856f9af10e14 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:21.144667818 +0000 UTC m=+41.566734632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls") pod "image-registry-8577649d7d-4btcd" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14") : secret "image-registry-tls" not found
Jul 24 01:30:19.918795 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:19.918750    2564 generic.go:356] "Generic (PLEG): container finished" podID="b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5" containerID="e97ad0617c53cc5852a73b42f3388ffd07c92114851c34aafa979f2aaf966383" exitCode=0
Jul 24 01:30:19.919407 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:19.918816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerDied","Data":"e97ad0617c53cc5852a73b42f3388ffd07c92114851c34aafa979f2aaf966383"}
Jul 24 01:30:20.925679 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:20.925583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tj678" event={"ID":"b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5","Type":"ContainerStarted","Data":"7bfa73e508e0797691b87484856821695f8524636164364766921ccd40e02883"}
Jul 24 01:30:20.949397 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:20.949340    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tj678" podStartSLOduration=7.501002632 podStartE2EDuration="37.949319393s" totalImagesPullingTime="30.448316761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:20.947149741 +0000 UTC m=+41.369216597" watchObservedRunningTime="2026-07-24 01:30:20.949319393 +0000 UTC m=+41.371386231"
Jul 24 01:30:21.158800 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.158764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:21.158963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.158831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:21.158963 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:21.158922    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 01:30:21.158963 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:21.158943    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8577649d7d-4btcd: secret "image-registry-tls" not found
Jul 24 01:30:21.158963 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:21.158927    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 01:30:21.159128 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:21.158995    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls podName:432593c2-05d4-4dfc-8246-87cc5b5d91c5 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:25.158982685 +0000 UTC m=+45.581049499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls") pod "dns-default-z6ckz" (UID: "432593c2-05d4-4dfc-8246-87cc5b5d91c5") : secret "dns-default-metrics-tls" not found
Jul 24 01:30:21.159128 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:21.159009    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls podName:ae8f6899-df6c-445c-98b4-856f9af10e14 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:25.15900329 +0000 UTC m=+45.581070149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls") pod "image-registry-8577649d7d-4btcd" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14") : secret "image-registry-tls" not found
Jul 24 01:30:21.762370 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.762336    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"]
Jul 24 01:30:21.780291 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.780247    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"]
Jul 24 01:30:21.780461 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.780391    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"
Jul 24 01:30:21.782764 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.782740    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 01:30:21.782913 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.782739    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bxsjk\""
Jul 24 01:30:21.782913 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.782740    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:30:21.802363 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.802335    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ppgdx"]
Jul 24 01:30:21.802474 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.802446    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"
Jul 24 01:30:21.805000 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.804969    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q54tj\""
Jul 24 01:30:21.828431 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.828404    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-4rr9p"]
Jul 24 01:30:21.828555 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.828530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.831125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.831100    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:30:21.831125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.831104    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 24 01:30:21.831321 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.831146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 24 01:30:21.831321 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.831295    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9c74x\""
Jul 24 01:30:21.831321 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.831316    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 24 01:30:21.835210 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.835193    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 24 01:30:21.846472 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.846450    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"]
Jul 24 01:30:21.846472 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.846480    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"]
Jul 24 01:30:21.846631 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.846496    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-4rr9p"]
Jul 24 01:30:21.846631 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.846513    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ppgdx"]
Jul 24 01:30:21.846631 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.846583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.849148 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.849120    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 24 01:30:21.849259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.849147    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gh5wt\""
Jul 24 01:30:21.849577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.849549    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 01:30:21.849651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.849632    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 24 01:30:21.851224 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.851210    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 01:30:21.857116 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.857094    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 24 01:30:21.863644 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2tf\" (UniqueName: \"kubernetes.io/projected/1f35de79-075f-4d52-b979-c0d51c69722d-kube-api-access-9d2tf\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.863745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f35de79-075f-4d52-b979-c0d51c69722d-serving-cert\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.863745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863680    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f35de79-075f-4d52-b979-c0d51c69722d-tmp\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.863745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f35de79-075f-4d52-b979-c0d51c69722d-service-ca-bundle\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.863861 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f35de79-075f-4d52-b979-c0d51c69722d-snapshots\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.863861 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqsd\" (UniqueName: \"kubernetes.io/projected/91ada225-88ab-4c5c-81b7-a67d9d3537af-kube-api-access-nvqsd\") pod \"network-check-source-7d88cb55d8-25r79\" (UID: \"91ada225-88ab-4c5c-81b7-a67d9d3537af\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"
Jul 24 01:30:21.863971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxqv\" (UniqueName: \"kubernetes.io/projected/1325aa34-e514-4dc1-9f59-11baa8109142-kube-api-access-zdxqv\") pod \"volume-data-source-validator-549c6d54cf-czjnh\" (UID: \"1325aa34-e514-4dc1-9f59-11baa8109142\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"
Jul 24 01:30:21.863971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9f43754-34d2-4c03-8535-5b4b5208db5c-config\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.863971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9f43754-34d2-4c03-8535-5b4b5208db5c-serving-cert\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.863971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9f43754-34d2-4c03-8535-5b4b5208db5c-trusted-ca\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.863971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.863967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnjz\" (UniqueName: \"kubernetes.io/projected/f9f43754-34d2-4c03-8535-5b4b5208db5c-kube-api-access-vfnjz\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.864199 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.864053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f35de79-075f-4d52-b979-c0d51c69722d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.929962 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.929922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qj8mc" event={"ID":"797c9bee-eab6-41f0-a52a-e64cd1034f1f","Type":"ContainerStarted","Data":"4b1e5c6fa6901d82496ccadd7c955857a62fb41d9465fbe70ab6f953ea60a974"}
Jul 24 01:30:21.946273 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.946236    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"]
Jul 24 01:30:21.948400 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.948353    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qj8mc" podStartSLOduration=2.186183363 podStartE2EDuration="4.948339021s" totalImagesPullingTime="2.762155658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:21.946708573 +0000 UTC m=+42.368775406" watchObservedRunningTime="2026-07-24 01:30:21.948339021 +0000 UTC m=+42.370405835"
Jul 24 01:30:21.964551 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.964524    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"]
Jul 24 01:30:21.964694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.964642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.964817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f35de79-075f-4d52-b979-c0d51c69722d-tmp\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.964851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f35de79-075f-4d52-b979-c0d51c69722d-service-ca-bundle\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.964962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f35de79-075f-4d52-b979-c0d51c69722d-snapshots\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.964987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqsd\" (UniqueName: \"kubernetes.io/projected/91ada225-88ab-4c5c-81b7-a67d9d3537af-kube-api-access-nvqsd\") pod \"network-check-source-7d88cb55d8-25r79\" (UID: \"91ada225-88ab-4c5c-81b7-a67d9d3537af\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxqv\" (UniqueName: \"kubernetes.io/projected/1325aa34-e514-4dc1-9f59-11baa8109142-kube-api-access-zdxqv\") pod \"volume-data-source-validator-549c6d54cf-czjnh\" (UID: \"1325aa34-e514-4dc1-9f59-11baa8109142\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9f43754-34d2-4c03-8535-5b4b5208db5c-config\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9f43754-34d2-4c03-8535-5b4b5208db5c-serving-cert\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9f43754-34d2-4c03-8535-5b4b5208db5c-trusted-ca\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnjz\" (UniqueName: \"kubernetes.io/projected/f9f43754-34d2-4c03-8535-5b4b5208db5c-kube-api-access-vfnjz\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f35de79-075f-4d52-b979-c0d51c69722d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965279    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2tf\" (UniqueName: \"kubernetes.io/projected/1f35de79-075f-4d52-b979-c0d51c69722d-kube-api-access-9d2tf\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f35de79-075f-4d52-b979-c0d51c69722d-tmp\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f35de79-075f-4d52-b979-c0d51c69722d-serving-cert\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.965622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.965601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f35de79-075f-4d52-b979-c0d51c69722d-service-ca-bundle\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.966355 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.966157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f35de79-075f-4d52-b979-c0d51c69722d-snapshots\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.966355 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.966204    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9f43754-34d2-4c03-8535-5b4b5208db5c-config\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.966969 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.966944    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f35de79-075f-4d52-b979-c0d51c69722d-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.967123 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.966979    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9f43754-34d2-4c03-8535-5b4b5208db5c-trusted-ca\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.967407 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.967381    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 01:30:21.967543 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.967391    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:30:21.967609 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.967449    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dg5zp\""
Jul 24 01:30:21.967960 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.967942    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 01:30:21.969406 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.969386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f35de79-075f-4d52-b979-c0d51c69722d-serving-cert\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.969478 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.969388    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9f43754-34d2-4c03-8535-5b4b5208db5c-serving-cert\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.975284 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.975243    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnjz\" (UniqueName: \"kubernetes.io/projected/f9f43754-34d2-4c03-8535-5b4b5208db5c-kube-api-access-vfnjz\") pod \"console-operator-584dcb6b54-ppgdx\" (UID: \"f9f43754-34d2-4c03-8535-5b4b5208db5c\") " pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:21.975380 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.975312    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqsd\" (UniqueName: \"kubernetes.io/projected/91ada225-88ab-4c5c-81b7-a67d9d3537af-kube-api-access-nvqsd\") pod \"network-check-source-7d88cb55d8-25r79\" (UID: \"91ada225-88ab-4c5c-81b7-a67d9d3537af\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"
Jul 24 01:30:21.975612 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.975590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2tf\" (UniqueName: \"kubernetes.io/projected/1f35de79-075f-4d52-b979-c0d51c69722d-kube-api-access-9d2tf\") pod \"insights-operator-5885fdb759-4rr9p\" (UID: \"1f35de79-075f-4d52-b979-c0d51c69722d\") " pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:21.976216 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.976194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxqv\" (UniqueName: \"kubernetes.io/projected/1325aa34-e514-4dc1-9f59-11baa8109142-kube-api-access-zdxqv\") pod \"volume-data-source-validator-549c6d54cf-czjnh\" (UID: \"1325aa34-e514-4dc1-9f59-11baa8109142\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"
Jul 24 01:30:21.991672 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.991651    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"]
Jul 24 01:30:21.991792 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.991775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:21.994019 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.993983    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sjz9w\""
Jul 24 01:30:21.994019 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.994011    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 01:30:21.994180 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:21.994042    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 01:30:22.011899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.011874    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"]
Jul 24 01:30:22.012012 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.011908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"]
Jul 24 01:30:22.012012 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.011989    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.012107 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.012043    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"]
Jul 24 01:30:22.014293 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.014245    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 01:30:22.014490 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.014477    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qq7mb\""
Jul 24 01:30:22.014728 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.014707    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 01:30:22.014789 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.014774    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 01:30:22.014837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.014827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:30:22.065744 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e0f32d1-116e-47b7-be49-9928e551ca9e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:22.065897 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065778    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:22.065897 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a755877-b146-4230-8b54-d66daa557182-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.065897 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxg5\" (UniqueName: \"kubernetes.io/projected/feb4075f-47a5-4105-b7c0-ec7beb8801e6-kube-api-access-bvxg5\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:22.066018 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:22.066018 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrj2w\" (UniqueName: \"kubernetes.io/projected/7a755877-b146-4230-8b54-d66daa557182-kube-api-access-nrj2w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.066018 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.065978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a755877-b146-4230-8b54-d66daa557182-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.089720 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.089700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"
Jul 24 01:30:22.110425 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.110403    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"
Jul 24 01:30:22.136733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.136699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:22.155000 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.154974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-4rr9p"
Jul 24 01:30:22.167403 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxg5\" (UniqueName: \"kubernetes.io/projected/feb4075f-47a5-4105-b7c0-ec7beb8801e6-kube-api-access-bvxg5\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:22.167509 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:22.167509 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrj2w\" (UniqueName: \"kubernetes.io/projected/7a755877-b146-4230-8b54-d66daa557182-kube-api-access-nrj2w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.167578 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a755877-b146-4230-8b54-d66daa557182-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.167611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e0f32d1-116e-47b7-be49-9928e551ca9e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:22.167663 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:22.167697 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.167680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a755877-b146-4230-8b54-d66daa557182-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.168699 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.168289    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 01:30:22.168699 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.168359    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert podName:3e0f32d1-116e-47b7-be49-9928e551ca9e nodeName:}" failed. No retries permitted until 2026-07-24 01:30:22.66833854 +0000 UTC m=+43.090405354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rcnft" (UID: "3e0f32d1-116e-47b7-be49-9928e551ca9e") : secret "networking-console-plugin-cert" not found
Jul 24 01:30:22.168699 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.168543    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 01:30:22.168699 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.168589    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls podName:feb4075f-47a5-4105-b7c0-ec7beb8801e6 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:22.668574356 +0000 UTC m=+43.090641174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls") pod "cluster-samples-operator-6748489656-p6zwg" (UID: "feb4075f-47a5-4105-b7c0-ec7beb8801e6") : secret "samples-operator-tls" not found
Jul 24 01:30:22.168974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.168732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e0f32d1-116e-47b7-be49-9928e551ca9e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:22.169054 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.168968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a755877-b146-4230-8b54-d66daa557182-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.171040 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.170999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a755877-b146-4230-8b54-d66daa557182-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.178081 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.177904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxg5\" (UniqueName: \"kubernetes.io/projected/feb4075f-47a5-4105-b7c0-ec7beb8801e6-kube-api-access-bvxg5\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:22.178389 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.178363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrj2w\" (UniqueName: \"kubernetes.io/projected/7a755877-b146-4230-8b54-d66daa557182-kube-api-access-nrj2w\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-lq465\" (UID: \"7a755877-b146-4230-8b54-d66daa557182\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.266814 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.264015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh"]
Jul 24 01:30:22.312748 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.312715    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79"]
Jul 24 01:30:22.320393 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.320182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"
Jul 24 01:30:22.349050 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.349000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ppgdx"]
Jul 24 01:30:22.352152 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:22.352122    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f43754_34d2_4c03_8535_5b4b5208db5c.slice/crio-e2283998e1224c62c2f5bec0ac0c028d4dd76cd00749c7ece9c3c2a0af766e8a WatchSource:0}: Error finding container e2283998e1224c62c2f5bec0ac0c028d4dd76cd00749c7ece9c3c2a0af766e8a: Status 404 returned error can't find the container with id e2283998e1224c62c2f5bec0ac0c028d4dd76cd00749c7ece9c3c2a0af766e8a
Jul 24 01:30:22.371558 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.371529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-4rr9p"]
Jul 24 01:30:22.374542 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:22.374513    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f35de79_075f_4d52_b979_c0d51c69722d.slice/crio-f2d0e5454657630924561e67d8193609abd55d694222569e2ebef1dcd84e6d9c WatchSource:0}: Error finding container f2d0e5454657630924561e67d8193609abd55d694222569e2ebef1dcd84e6d9c: Status 404 returned error can't find the container with id f2d0e5454657630924561e67d8193609abd55d694222569e2ebef1dcd84e6d9c
Jul 24 01:30:22.455831 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.455797    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465"]
Jul 24 01:30:22.463925 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:22.463896    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a755877_b146_4230_8b54_d66daa557182.slice/crio-8c0cafea546c86c6e98e071d7d0dfaec95703f4cbe4cc56a352e0972cf69914e WatchSource:0}: Error finding container 8c0cafea546c86c6e98e071d7d0dfaec95703f4cbe4cc56a352e0972cf69914e: Status 404 returned error can't find the container with id 8c0cafea546c86c6e98e071d7d0dfaec95703f4cbe4cc56a352e0972cf69914e
Jul 24 01:30:22.671719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.671629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:22.671719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.671696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:22.671973 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.671805    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 01:30:22.671973 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.671808    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 01:30:22.671973 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.671866    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert podName:3e0f32d1-116e-47b7-be49-9928e551ca9e nodeName:}" failed. No retries permitted until 2026-07-24 01:30:23.671848683 +0000 UTC m=+44.093915500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rcnft" (UID: "3e0f32d1-116e-47b7-be49-9928e551ca9e") : secret "networking-console-plugin-cert" not found
Jul 24 01:30:22.671973 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:22.671881    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls podName:feb4075f-47a5-4105-b7c0-ec7beb8801e6 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:23.671874197 +0000 UTC m=+44.093941011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls") pod "cluster-samples-operator-6748489656-p6zwg" (UID: "feb4075f-47a5-4105-b7c0-ec7beb8801e6") : secret "samples-operator-tls" not found
Jul 24 01:30:22.934067 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.933962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79" event={"ID":"91ada225-88ab-4c5c-81b7-a67d9d3537af","Type":"ContainerStarted","Data":"ab02988f1873974c5fcb44fd2f4e653f90bca3d51e4502964f98f85d807a2e5b"}
Jul 24 01:30:22.935372 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.935346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh" event={"ID":"1325aa34-e514-4dc1-9f59-11baa8109142","Type":"ContainerStarted","Data":"d02299c4b75dabfb1b8b874cce38553fd0a3566ecb10f001db59a8157eac95a2"}
Jul 24 01:30:22.936839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.936800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-4rr9p" event={"ID":"1f35de79-075f-4d52-b979-c0d51c69722d","Type":"ContainerStarted","Data":"f2d0e5454657630924561e67d8193609abd55d694222569e2ebef1dcd84e6d9c"}
Jul 24 01:30:22.938161 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.938125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465" event={"ID":"7a755877-b146-4230-8b54-d66daa557182","Type":"ContainerStarted","Data":"8c0cafea546c86c6e98e071d7d0dfaec95703f4cbe4cc56a352e0972cf69914e"}
Jul 24 01:30:22.939356 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:22.939217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" event={"ID":"f9f43754-34d2-4c03-8535-5b4b5208db5c","Type":"ContainerStarted","Data":"e2283998e1224c62c2f5bec0ac0c028d4dd76cd00749c7ece9c3c2a0af766e8a"}
Jul 24 01:30:23.682269 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:23.681195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:23.682269 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:23.681303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:23.682269 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:23.681573    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 01:30:23.682269 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:23.681659    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert podName:3e0f32d1-116e-47b7-be49-9928e551ca9e nodeName:}" failed. No retries permitted until 2026-07-24 01:30:25.681638194 +0000 UTC m=+46.103705011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rcnft" (UID: "3e0f32d1-116e-47b7-be49-9928e551ca9e") : secret "networking-console-plugin-cert" not found
Jul 24 01:30:23.682269 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:23.682120    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 01:30:23.682269 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:23.682227    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls podName:feb4075f-47a5-4105-b7c0-ec7beb8801e6 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:25.68215987 +0000 UTC m=+46.104226689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls") pod "cluster-samples-operator-6748489656-p6zwg" (UID: "feb4075f-47a5-4105-b7c0-ec7beb8801e6") : secret "samples-operator-tls" not found
Jul 24 01:30:25.199644 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:25.199602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:25.200113 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:25.199670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:25.200113 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.199787    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 01:30:25.200113 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.199807    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 01:30:25.200113 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.199814    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8577649d7d-4btcd: secret "image-registry-tls" not found
Jul 24 01:30:25.200113 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.199875    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls podName:432593c2-05d4-4dfc-8246-87cc5b5d91c5 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:33.199856374 +0000 UTC m=+53.621923212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls") pod "dns-default-z6ckz" (UID: "432593c2-05d4-4dfc-8246-87cc5b5d91c5") : secret "dns-default-metrics-tls" not found
Jul 24 01:30:25.200113 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.199896    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls podName:ae8f6899-df6c-445c-98b4-856f9af10e14 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:33.199886634 +0000 UTC m=+53.621953465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls") pod "image-registry-8577649d7d-4btcd" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14") : secret "image-registry-tls" not found
Jul 24 01:30:25.704428 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:25.704390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:25.704602 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:25.704510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:25.704602 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.704559    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 01:30:25.704748 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.704608    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 01:30:25.704748 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.704628    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert podName:3e0f32d1-116e-47b7-be49-9928e551ca9e nodeName:}" failed. No retries permitted until 2026-07-24 01:30:29.704611115 +0000 UTC m=+50.126677928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rcnft" (UID: "3e0f32d1-116e-47b7-be49-9928e551ca9e") : secret "networking-console-plugin-cert" not found
Jul 24 01:30:25.704748 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:25.704656    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls podName:feb4075f-47a5-4105-b7c0-ec7beb8801e6 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:29.70464075 +0000 UTC m=+50.126707580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls") pod "cluster-samples-operator-6748489656-p6zwg" (UID: "feb4075f-47a5-4105-b7c0-ec7beb8801e6") : secret "samples-operator-tls" not found
Jul 24 01:30:27.956245 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:27.956216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh" event={"ID":"1325aa34-e514-4dc1-9f59-11baa8109142","Type":"ContainerStarted","Data":"649e2741ac043fc0d8cc781ee940d3f0ba8dee68ca11302278cb0d0d11010838"}
Jul 24 01:30:27.957654 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:27.957626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" event={"ID":"f9f43754-34d2-4c03-8535-5b4b5208db5c","Type":"ContainerStarted","Data":"1a50030fc976197b4b8779b3ba158bb55217b430388d8c1fe97abafb010a634b"}
Jul 24 01:30:28.962440 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.962399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79" event={"ID":"91ada225-88ab-4c5c-81b7-a67d9d3537af","Type":"ContainerStarted","Data":"13e55cfcf0ff4f9d94d76ddd2ab87581676f98960a6ac1800080d2600739839b"}
Jul 24 01:30:28.964016 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.963986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-4rr9p" event={"ID":"1f35de79-075f-4d52-b979-c0d51c69722d","Type":"ContainerStarted","Data":"2697351412c2cd2497499dd59b0b5ca3b296523a6701fb2298ade406a10bbc7b"}
Jul 24 01:30:28.965521 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.965500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465" event={"ID":"7a755877-b146-4230-8b54-d66daa557182","Type":"ContainerStarted","Data":"ae42a568194617b631c66ebc337394d34a32490e443a27ae6ff23d3167490d79"}
Jul 24 01:30:28.967136 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.967118    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/0.log"
Jul 24 01:30:28.967226 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.967154    2564 generic.go:356] "Generic (PLEG): container finished" podID="f9f43754-34d2-4c03-8535-5b4b5208db5c" containerID="1a50030fc976197b4b8779b3ba158bb55217b430388d8c1fe97abafb010a634b" exitCode=255
Jul 24 01:30:28.967300 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.967234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" event={"ID":"f9f43754-34d2-4c03-8535-5b4b5208db5c","Type":"ContainerDied","Data":"1a50030fc976197b4b8779b3ba158bb55217b430388d8c1fe97abafb010a634b"}
Jul 24 01:30:28.967370 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.967356    2564 scope.go:117] "RemoveContainer" containerID="1a50030fc976197b4b8779b3ba158bb55217b430388d8c1fe97abafb010a634b"
Jul 24 01:30:28.982367 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.982307    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-25r79" podStartSLOduration=2.477195444 podStartE2EDuration="7.982286784s" totalImagesPullingTime="5.50509134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:28.980263976 +0000 UTC m=+49.402330814" watchObservedRunningTime="2026-07-24 01:30:28.982286784 +0000 UTC m=+49.404353685"
Jul 24 01:30:28.997859 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:28.997801    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465" podStartSLOduration=2.639330041 podStartE2EDuration="7.997779982s" totalImagesPullingTime="5.358449941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:28.9969246 +0000 UTC m=+49.418991438" watchObservedRunningTime="2026-07-24 01:30:28.997779982 +0000 UTC m=+49.419846817"
Jul 24 01:30:29.035340 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.035276    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-czjnh" podStartSLOduration=2.481181233 podStartE2EDuration="8.035258945s" totalImagesPullingTime="5.554077712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:29.034357535 +0000 UTC m=+49.456424372" watchObservedRunningTime="2026-07-24 01:30:29.035258945 +0000 UTC m=+49.457325785"
Jul 24 01:30:29.055991 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.055891    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-4rr9p" podStartSLOduration=2.602075757 podStartE2EDuration="8.055874749s" totalImagesPullingTime="5.453798992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:29.055392451 +0000 UTC m=+49.477459312" watchObservedRunningTime="2026-07-24 01:30:29.055874749 +0000 UTC m=+49.477941584"
Jul 24 01:30:29.572687 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.572649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2dpdk"]
Jul 24 01:30:29.598857 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.598828    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2dpdk"]
Jul 24 01:30:29.599064 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.598950    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.601174 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.601151    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 01:30:29.601394 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.601377    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pds6h\""
Jul 24 01:30:29.601662 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.601633    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 01:30:29.637910 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.637881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4d617d-edca-4b97-a24a-23449fe67b82-data-volume\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.638100 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.637954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lbk\" (UniqueName: \"kubernetes.io/projected/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-api-access-d6lbk\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.638100 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.637983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb4d617d-edca-4b97-a24a-23449fe67b82-crio-socket\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.638100 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.638000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.638100 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.638058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739351 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739524 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:29.739524 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4d617d-edca-4b97-a24a-23449fe67b82-data-volume\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739524 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:29.739524 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739467    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:29.739524 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lbk\" (UniqueName: \"kubernetes.io/projected/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-api-access-d6lbk\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739524 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739517    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb4d617d-edca-4b97-a24a-23449fe67b82-crio-socket\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739566    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls podName:feb4075f-47a5-4105-b7c0-ec7beb8801e6 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:37.739551407 +0000 UTC m=+58.161618221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls") pod "cluster-samples-operator-6748489656-p6zwg" (UID: "feb4075f-47a5-4105-b7c0-ec7beb8801e6") : secret "samples-operator-tls" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739606    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739654    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:30.239639146 +0000 UTC m=+50.661705990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : configmap "kube-rbac-proxy" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739668    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:30.239660269 +0000 UTC m=+50.661727083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739677    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739708    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb4d617d-edca-4b97-a24a-23449fe67b82-crio-socket\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.739723    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert podName:3e0f32d1-116e-47b7-be49-9928e551ca9e nodeName:}" failed. No retries permitted until 2026-07-24 01:30:37.73971249 +0000 UTC m=+58.161779308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rcnft" (UID: "3e0f32d1-116e-47b7-be49-9928e551ca9e") : secret "networking-console-plugin-cert" not found
Jul 24 01:30:29.739793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.739776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4d617d-edca-4b97-a24a-23449fe67b82-data-volume\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.747069 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.747047    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lbk\" (UniqueName: \"kubernetes.io/projected/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-api-access-d6lbk\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:29.975594 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.975565    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:30:29.976053 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.976019    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/0.log"
Jul 24 01:30:29.976109 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.976083    2564 generic.go:356] "Generic (PLEG): container finished" podID="f9f43754-34d2-4c03-8535-5b4b5208db5c" containerID="b4cf0455ffc4a93677b709644886a43acadff7122268a6cc6e01bbae2f162ba1" exitCode=255
Jul 24 01:30:29.976194 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.976168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" event={"ID":"f9f43754-34d2-4c03-8535-5b4b5208db5c","Type":"ContainerDied","Data":"b4cf0455ffc4a93677b709644886a43acadff7122268a6cc6e01bbae2f162ba1"}
Jul 24 01:30:29.976283 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.976269    2564 scope.go:117] "RemoveContainer" containerID="1a50030fc976197b4b8779b3ba158bb55217b430388d8c1fe97abafb010a634b"
Jul 24 01:30:29.976601 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:29.976546    2564 scope.go:117] "RemoveContainer" containerID="b4cf0455ffc4a93677b709644886a43acadff7122268a6cc6e01bbae2f162ba1"
Jul 24 01:30:29.976753 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:29.976729    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ppgdx_openshift-console-operator(f9f43754-34d2-4c03-8535-5b4b5208db5c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" podUID="f9f43754-34d2-4c03-8535-5b4b5208db5c"
Jul 24 01:30:30.243942 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:30.243839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:30.244104 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:30.244003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:30.244104 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:30.244014    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:30.244104 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:30.244098    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:31.244078524 +0000 UTC m=+51.666145352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:30.244229 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:30.244121    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 01:30:30.244229 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:30.244175    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:31.244162357 +0000 UTC m=+51.666229171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : configmap "kube-rbac-proxy" not found
Jul 24 01:30:30.981471 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:30.981445    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:30:30.981821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:30.981779    2564 scope.go:117] "RemoveContainer" containerID="b4cf0455ffc4a93677b709644886a43acadff7122268a6cc6e01bbae2f162ba1"
Jul 24 01:30:30.981964 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:30.981945    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ppgdx_openshift-console-operator(f9f43754-34d2-4c03-8535-5b4b5208db5c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" podUID="f9f43754-34d2-4c03-8535-5b4b5208db5c"
Jul 24 01:30:31.252704 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:31.252605    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:31.252704 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:31.252670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:31.252893 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:31.252739    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 01:30:31.252893 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:31.252762    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:31.252893 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:31.252820    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:33.252802075 +0000 UTC m=+53.674868908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:31.252893 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:31.252835    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:33.252829137 +0000 UTC m=+53.674895952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : configmap "kube-rbac-proxy" not found
Jul 24 01:30:32.136832 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:32.136803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:32.137323 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:32.136861    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:32.137323 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:32.137294    2564 scope.go:117] "RemoveContainer" containerID="b4cf0455ffc4a93677b709644886a43acadff7122268a6cc6e01bbae2f162ba1"
Jul 24 01:30:32.137504 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:32.137480    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ppgdx_openshift-console-operator(f9f43754-34d2-4c03-8535-5b4b5208db5c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" podUID="f9f43754-34d2-4c03-8535-5b4b5208db5c"
Jul 24 01:30:32.371714 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:32.371686    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r8dkq_25e37d82-6241-46f1-974c-655e53ab2189/dns-node-resolver/0.log"
Jul 24 01:30:33.168378 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:33.168347    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4dtpj_c132a9e1-eb6e-4602-bfa2-40af3a44a8d6/node-ca/0.log"
Jul 24 01:30:33.271866 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:33.271823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:33.271880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:33.271913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:33.271951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.271984    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272012    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8577649d7d-4btcd: secret "image-registry-tls" not found
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272019    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 01:30:33.272083 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272085    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 01:30:33.272355 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272104    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:33.272355 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272088    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls podName:ae8f6899-df6c-445c-98b4-856f9af10e14 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:49.272071602 +0000 UTC m=+69.694138420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls") pod "image-registry-8577649d7d-4btcd" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14") : secret "image-registry-tls" not found
Jul 24 01:30:33.272355 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272149    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:37.272136525 +0000 UTC m=+57.694203340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : configmap "kube-rbac-proxy" not found
Jul 24 01:30:33.272355 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272166    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls podName:432593c2-05d4-4dfc-8246-87cc5b5d91c5 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:49.272160085 +0000 UTC m=+69.694226898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls") pod "dns-default-z6ckz" (UID: "432593c2-05d4-4dfc-8246-87cc5b5d91c5") : secret "dns-default-metrics-tls" not found
Jul 24 01:30:33.272355 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:33.272179    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:37.272174318 +0000 UTC m=+57.694241131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:34.368041 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:34.367986    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qj8mc_797c9bee-eab6-41f0-a52a-e64cd1034f1f/serve-healthcheck-canary/0.log"
Jul 24 01:30:34.969923 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:34.969893    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-lq465_7a755877-b146-4230-8b54-d66daa557182/kube-storage-version-migrator-operator/0.log"
Jul 24 01:30:37.302755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.302721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:37.303271 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.302773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:37.303271 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:37.302969    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:37.303271 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:37.303069    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls podName:cb4d617d-edca-4b97-a24a-23449fe67b82 nodeName:}" failed. No retries permitted until 2026-07-24 01:30:45.303019365 +0000 UTC m=+65.725086180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2dpdk" (UID: "cb4d617d-edca-4b97-a24a-23449fe67b82") : secret "insights-runtime-extractor-tls" not found
Jul 24 01:30:37.303515 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.303491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb4d617d-edca-4b97-a24a-23449fe67b82-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:37.807248 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.807210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:37.807435 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.807293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:37.807435 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:37.807426    2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 01:30:37.807527 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:30:37.807491    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert podName:3e0f32d1-116e-47b7-be49-9928e551ca9e nodeName:}" failed. No retries permitted until 2026-07-24 01:30:53.807473773 +0000 UTC m=+74.229540587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-rcnft" (UID: "3e0f32d1-116e-47b7-be49-9928e551ca9e") : secret "networking-console-plugin-cert" not found
Jul 24 01:30:37.809620 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.809596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/feb4075f-47a5-4105-b7c0-ec7beb8801e6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-p6zwg\" (UID: \"feb4075f-47a5-4105-b7c0-ec7beb8801e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:37.889493 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:37.889458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"
Jul 24 01:30:38.033521 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:38.033488    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg"]
Jul 24 01:30:39.007000 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:39.006952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg" event={"ID":"feb4075f-47a5-4105-b7c0-ec7beb8801e6","Type":"ContainerStarted","Data":"ab1ee04e8a8e11aac450bd995789813ade927ecbf0d76ba814e14712e849da6a"}
Jul 24 01:30:40.012598 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:40.012561    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg" event={"ID":"feb4075f-47a5-4105-b7c0-ec7beb8801e6","Type":"ContainerStarted","Data":"49c286ecbca932fece01ff0fa8517acc7c3c32bfdc244e7cf365d371d07ac36a"}
Jul 24 01:30:40.013081 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:40.012609    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg" event={"ID":"feb4075f-47a5-4105-b7c0-ec7beb8801e6","Type":"ContainerStarted","Data":"4706e89daeaee46d718322b8bff7e7e75f8c7151abc1675fbbf2e1fe4ce1bb88"}
Jul 24 01:30:40.028186 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:40.028140    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-p6zwg" podStartSLOduration=17.585815986 podStartE2EDuration="19.028124379s" totalImagesPullingTime="1.442308393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:40.02769655 +0000 UTC m=+60.449763387" watchObservedRunningTime="2026-07-24 01:30:40.028124379 +0000 UTC m=+60.450191215"
Jul 24 01:30:42.902813 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:42.902776    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-85shc"
Jul 24 01:30:45.203781 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:45.203743    2564 scope.go:117] "RemoveContainer" containerID="b4cf0455ffc4a93677b709644886a43acadff7122268a6cc6e01bbae2f162ba1"
Jul 24 01:30:45.366898 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:45.366819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:45.369350 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:45.369318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb4d617d-edca-4b97-a24a-23449fe67b82-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2dpdk\" (UID: \"cb4d617d-edca-4b97-a24a-23449fe67b82\") " pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:45.510395 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:45.510313    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pds6h\""
Jul 24 01:30:45.519170 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:45.519144    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2dpdk"
Jul 24 01:30:45.650992 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:45.650956    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2dpdk"]
Jul 24 01:30:45.654128 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:45.654090    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4d617d_edca_4b97_a24a_23449fe67b82.slice/crio-7b90e4b06f6a00dbfba96015da258c678db9592c21a657ecea33ddcd5ccf76eb WatchSource:0}: Error finding container 7b90e4b06f6a00dbfba96015da258c678db9592c21a657ecea33ddcd5ccf76eb: Status 404 returned error can't find the container with id 7b90e4b06f6a00dbfba96015da258c678db9592c21a657ecea33ddcd5ccf76eb
Jul 24 01:30:46.035734 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:46.035705    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:30:46.035923 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:46.035806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" event={"ID":"f9f43754-34d2-4c03-8535-5b4b5208db5c","Type":"ContainerStarted","Data":"6edabd308bd1ef76da8d72cf009bc5e4283233f0f8d484ebea5d3f941301410b"}
Jul 24 01:30:46.036125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:46.036097    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:46.037555 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:46.037528    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2dpdk" event={"ID":"cb4d617d-edca-4b97-a24a-23449fe67b82","Type":"ContainerStarted","Data":"7b90e4b06f6a00dbfba96015da258c678db9592c21a657ecea33ddcd5ccf76eb"}
Jul 24 01:30:46.050957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:46.050912    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx" podStartSLOduration=19.581391576 podStartE2EDuration="25.050895026s" totalImagesPullingTime="5.46950345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:46.050839601 +0000 UTC m=+66.472906441" watchObservedRunningTime="2026-07-24 01:30:46.050895026 +0000 UTC m=+66.472961862"
Jul 24 01:30:46.353064 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:46.352959    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-ppgdx"
Jul 24 01:30:47.689874 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.689832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:47.690365 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.689903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:47.692456 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.692436    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 01:30:47.693238 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.693220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 01:30:47.703363 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.703334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3d94894-5060-45d1-bd7b-1fea2cedeb43-metrics-certs\") pod \"network-metrics-daemon-crts2\" (UID: \"c3d94894-5060-45d1-bd7b-1fea2cedeb43\") " pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:47.703490 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.703404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53-original-pull-secret\") pod \"global-pull-secret-syncer-2dzp8\" (UID: \"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53\") " pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:47.791226 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.791182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:47.794224 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.794195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7db9v\" (UniqueName: \"kubernetes.io/projected/4934861c-f221-4082-b600-b6f14c26185b-kube-api-access-7db9v\") pod \"network-check-target-9kkdx\" (UID: \"4934861c-f221-4082-b600-b6f14c26185b\") " pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:47.917746 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.917713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c62r2\""
Jul 24 01:30:47.924795 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.924765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pnzwr\""
Jul 24 01:30:47.925793 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.925756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:47.928655 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.928635    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dzp8"
Jul 24 01:30:47.933354 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:47.933334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crts2"
Jul 24 01:30:49.306524 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.306487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:49.307004 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.306555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:49.309671 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.309642    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/432593c2-05d4-4dfc-8246-87cc5b5d91c5-metrics-tls\") pod \"dns-default-z6ckz\" (UID: \"432593c2-05d4-4dfc-8246-87cc5b5d91c5\") " pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:49.309671 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.309663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"image-registry-8577649d7d-4btcd\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") " pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:49.538626 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.538593    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ld5t9\""
Jul 24 01:30:49.546690 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.546617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:49.551606 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.551559    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8g76\""
Jul 24 01:30:49.559505 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:49.559432    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:30:50.454620 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.454582    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"]
Jul 24 01:30:50.491583 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.491550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"]
Jul 24 01:30:50.491732 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.491646    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.494187 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.494162    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 01:30:50.494304 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.494162    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 01:30:50.494304 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.494167    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 01:30:50.495150 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.495135    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 01:30:50.617720 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.617590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8db79f07-321c-4cd6-b5ee-3876d138bc40-klusterlet-config\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.617720 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.617647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q49g4\" (UniqueName: \"kubernetes.io/projected/8db79f07-321c-4cd6-b5ee-3876d138bc40-kube-api-access-q49g4\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.617947 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.617730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8db79f07-321c-4cd6-b5ee-3876d138bc40-tmp\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.649107 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.649066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z6ckz"]
Jul 24 01:30:50.719130 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.719019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8db79f07-321c-4cd6-b5ee-3876d138bc40-tmp\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.719130 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.719117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8db79f07-321c-4cd6-b5ee-3876d138bc40-klusterlet-config\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.719324 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.719169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q49g4\" (UniqueName: \"kubernetes.io/projected/8db79f07-321c-4cd6-b5ee-3876d138bc40-kube-api-access-q49g4\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.719494 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.719469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8db79f07-321c-4cd6-b5ee-3876d138bc40-tmp\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.722016 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.721992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8db79f07-321c-4cd6-b5ee-3876d138bc40-klusterlet-config\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.729088 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.729061    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q49g4\" (UniqueName: \"kubernetes.io/projected/8db79f07-321c-4cd6-b5ee-3876d138bc40-kube-api-access-q49g4\") pod \"klusterlet-addon-workmgr-67c9f94c75-d9zbb\" (UID: \"8db79f07-321c-4cd6-b5ee-3876d138bc40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.801178 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:50.801130    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:30:50.957935 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:50.957892    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432593c2_05d4_4dfc_8246_87cc5b5d91c5.slice/crio-09fd976f3debec4849bc113860452a0c4cf285adeb53b6322678e1817798014c WatchSource:0}: Error finding container 09fd976f3debec4849bc113860452a0c4cf285adeb53b6322678e1817798014c: Status 404 returned error can't find the container with id 09fd976f3debec4849bc113860452a0c4cf285adeb53b6322678e1817798014c
Jul 24 01:30:51.078850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:51.078656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6ckz" event={"ID":"432593c2-05d4-4dfc-8246-87cc5b5d91c5","Type":"ContainerStarted","Data":"09fd976f3debec4849bc113860452a0c4cf285adeb53b6322678e1817798014c"}
Jul 24 01:30:51.148424 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:51.148392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2dzp8"]
Jul 24 01:30:51.211791 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:51.211765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"]
Jul 24 01:30:51.304583 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:51.304500    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9426fc_2fc6_41c7_a11b_51e7f2c1fb53.slice/crio-47c171ad4253408695f945343294ab88673299264c4744c57b0321ce7fda0ecd WatchSource:0}: Error finding container 47c171ad4253408695f945343294ab88673299264c4744c57b0321ce7fda0ecd: Status 404 returned error can't find the container with id 47c171ad4253408695f945343294ab88673299264c4744c57b0321ce7fda0ecd
Jul 24 01:30:51.305052 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:51.304970    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db79f07_321c_4cd6_b5ee_3876d138bc40.slice/crio-a9d2c42179136a8f2f2e7cf4c43c2598f16dfeba8b799586bed8adee73bc3c15 WatchSource:0}: Error finding container a9d2c42179136a8f2f2e7cf4c43c2598f16dfeba8b799586bed8adee73bc3c15: Status 404 returned error can't find the container with id a9d2c42179136a8f2f2e7cf4c43c2598f16dfeba8b799586bed8adee73bc3c15
Jul 24 01:30:51.377921 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:51.377889    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9kkdx"]
Jul 24 01:30:51.381195 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:51.381164    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crts2"]
Jul 24 01:30:51.382435 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:51.382352    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8577649d7d-4btcd"]
Jul 24 01:30:51.389404 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:51.389377    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4934861c_f221_4082_b600_b6f14c26185b.slice/crio-8824e7ae6df9636518fd9a9c1bd22d21744a1b7f4ace58c183aa7cc618f138a7 WatchSource:0}: Error finding container 8824e7ae6df9636518fd9a9c1bd22d21744a1b7f4ace58c183aa7cc618f138a7: Status 404 returned error can't find the container with id 8824e7ae6df9636518fd9a9c1bd22d21744a1b7f4ace58c183aa7cc618f138a7
Jul 24 01:30:51.389738 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:51.389712    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d94894_5060_45d1_bd7b_1fea2cedeb43.slice/crio-587be76aeb19c5abff63addff9af4d1ce96db4cd633b9bed3c88ac26d1059ccf WatchSource:0}: Error finding container 587be76aeb19c5abff63addff9af4d1ce96db4cd633b9bed3c88ac26d1059ccf: Status 404 returned error can't find the container with id 587be76aeb19c5abff63addff9af4d1ce96db4cd633b9bed3c88ac26d1059ccf
Jul 24 01:30:51.389971 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:30:51.389952    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8f6899_df6c_445c_98b4_856f9af10e14.slice/crio-baec91a63f93fc792b2be2917fc406764ea02f542f89b88ec92b207d7500ef5c WatchSource:0}: Error finding container baec91a63f93fc792b2be2917fc406764ea02f542f89b88ec92b207d7500ef5c: Status 404 returned error can't find the container with id baec91a63f93fc792b2be2917fc406764ea02f542f89b88ec92b207d7500ef5c
Jul 24 01:30:52.093733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.092732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9kkdx" event={"ID":"4934861c-f221-4082-b600-b6f14c26185b","Type":"ContainerStarted","Data":"014d1889e9b74b7c1a1b1ae240794dd25f820787c414e9357d883a6ebce3129a"}
Jul 24 01:30:52.093733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.092793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9kkdx" event={"ID":"4934861c-f221-4082-b600-b6f14c26185b","Type":"ContainerStarted","Data":"8824e7ae6df9636518fd9a9c1bd22d21744a1b7f4ace58c183aa7cc618f138a7"}
Jul 24 01:30:52.093733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.093653    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:30:52.097806 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.097617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb" event={"ID":"8db79f07-321c-4cd6-b5ee-3876d138bc40","Type":"ContainerStarted","Data":"a9d2c42179136a8f2f2e7cf4c43c2598f16dfeba8b799586bed8adee73bc3c15"}
Jul 24 01:30:52.101560 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.101440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crts2" event={"ID":"c3d94894-5060-45d1-bd7b-1fea2cedeb43","Type":"ContainerStarted","Data":"587be76aeb19c5abff63addff9af4d1ce96db4cd633b9bed3c88ac26d1059ccf"}
Jul 24 01:30:52.104880 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.104800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2dzp8" event={"ID":"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53","Type":"ContainerStarted","Data":"47c171ad4253408695f945343294ab88673299264c4744c57b0321ce7fda0ecd"}
Jul 24 01:30:52.111180 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.110620    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9kkdx" podStartSLOduration=70.110599165 podStartE2EDuration="1m10.110599165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:52.107878799 +0000 UTC m=+72.529945636" watchObservedRunningTime="2026-07-24 01:30:52.110599165 +0000 UTC m=+72.532666002"
Jul 24 01:30:52.115741 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.114678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8577649d7d-4btcd" event={"ID":"ae8f6899-df6c-445c-98b4-856f9af10e14","Type":"ContainerStarted","Data":"27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84"}
Jul 24 01:30:52.115741 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.114730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8577649d7d-4btcd" event={"ID":"ae8f6899-df6c-445c-98b4-856f9af10e14","Type":"ContainerStarted","Data":"baec91a63f93fc792b2be2917fc406764ea02f542f89b88ec92b207d7500ef5c"}
Jul 24 01:30:52.116450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.116407    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:30:52.122960 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.122891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2dpdk" event={"ID":"cb4d617d-edca-4b97-a24a-23449fe67b82","Type":"ContainerStarted","Data":"565549b5ec4e043f44313ab7e5b25732a83ac9bf269147e0fe4a3af2c233ef33"}
Jul 24 01:30:52.134896 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:52.134467    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8577649d7d-4btcd" podStartSLOduration=46.13444527 podStartE2EDuration="46.13444527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:30:52.133387442 +0000 UTC m=+72.555454270" watchObservedRunningTime="2026-07-24 01:30:52.13444527 +0000 UTC m=+72.556512107"
Jul 24 01:30:53.852821 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:53.852772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:53.856099 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:53.856043    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e0f32d1-116e-47b7-be49-9928e551ca9e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-rcnft\" (UID: \"3e0f32d1-116e-47b7-be49-9928e551ca9e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:54.103264 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:54.102955    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sjz9w\""
Jul 24 01:30:54.111071 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:54.110790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"
Jul 24 01:30:59.983707 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:30:59.983666    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-rcnft"]
Jul 24 01:31:00.048373 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:31:00.048328    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0f32d1_116e_47b7_be49_9928e551ca9e.slice/crio-14cf9c4c1abad8077d439884925532577c247f79a07b26909340ff1568c89410 WatchSource:0}: Error finding container 14cf9c4c1abad8077d439884925532577c247f79a07b26909340ff1568c89410: Status 404 returned error can't find the container with id 14cf9c4c1abad8077d439884925532577c247f79a07b26909340ff1568c89410
Jul 24 01:31:00.162948 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.162914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft" event={"ID":"3e0f32d1-116e-47b7-be49-9928e551ca9e","Type":"ContainerStarted","Data":"14cf9c4c1abad8077d439884925532577c247f79a07b26909340ff1568c89410"}
Jul 24 01:31:00.164586 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.164558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb" event={"ID":"8db79f07-321c-4cd6-b5ee-3876d138bc40","Type":"ContainerStarted","Data":"cc223007cd568e93692080a80ba4c3a5257a3ce00a2aef6241f853c4289e3335"}
Jul 24 01:31:00.164930 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.164909    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:31:00.166907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.166887    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb"
Jul 24 01:31:00.167387 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.167317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crts2" event={"ID":"c3d94894-5060-45d1-bd7b-1fea2cedeb43","Type":"ContainerStarted","Data":"c6db4bdf7ff2b5670ef2f6219079c22baa9887e161396c67a714fe209e5a3965"}
Jul 24 01:31:00.169051 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.168845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2dzp8" event={"ID":"ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53","Type":"ContainerStarted","Data":"097ca63251da9a23db19c0fc69db6e4a835bf259af11be7e33c2411f31119857"}
Jul 24 01:31:00.171603 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.171577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6ckz" event={"ID":"432593c2-05d4-4dfc-8246-87cc5b5d91c5","Type":"ContainerStarted","Data":"a28db0399ea21e31c3f6a88fb3a4a688e662eefc2549480e1f8155171b12e077"}
Jul 24 01:31:00.181403 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.181348    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67c9f94c75-d9zbb" podStartSLOduration=1.6558269559999998 podStartE2EDuration="10.181326485s" totalImagesPullingTime="8.525499529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:00.180583275 +0000 UTC m=+80.602650112" watchObservedRunningTime="2026-07-24 01:31:00.181326485 +0000 UTC m=+80.603393326"
Jul 24 01:31:00.207794 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:00.207742    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2dzp8" podStartSLOduration=68.703495123 podStartE2EDuration="1m17.207725571s" totalImagesPullingTime="8.504230448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:00.206855314 +0000 UTC m=+80.628922150" watchObservedRunningTime="2026-07-24 01:31:00.207725571 +0000 UTC m=+80.629792407"
Jul 24 01:31:01.178983 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:01.178913    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6ckz" event={"ID":"432593c2-05d4-4dfc-8246-87cc5b5d91c5","Type":"ContainerStarted","Data":"c9653025d6fb4d7d0deca186b6642dfa113885f800ec4ba0a6d7b5b710f555b6"}
Jul 24 01:31:01.179623 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:01.179059    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:31:01.181333 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:01.181304    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2dpdk" event={"ID":"cb4d617d-edca-4b97-a24a-23449fe67b82","Type":"ContainerStarted","Data":"fd51799434998205fcf8d789480b8ceb11d83c044861853ba8f36d6808abdf41"}
Jul 24 01:31:01.183516 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:01.183484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crts2" event={"ID":"c3d94894-5060-45d1-bd7b-1fea2cedeb43","Type":"ContainerStarted","Data":"d3c6fd102d39bd6bf1310f3726121caf7185f61654732d74dd2f1346ac07917b"}
Jul 24 01:31:01.197687 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:01.197627    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z6ckz" podStartSLOduration=41.781913834 podStartE2EDuration="44.197605164s" totalImagesPullingTime="2.41569133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:01.195277202 +0000 UTC m=+81.617344064" watchObservedRunningTime="2026-07-24 01:31:01.197605164 +0000 UTC m=+81.619672001"
Jul 24 01:31:01.211656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:01.211594    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-crts2" podStartSLOduration=69.769412858 podStartE2EDuration="1m18.211574731s" totalImagesPullingTime="8.442161873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:01.210718552 +0000 UTC m=+81.632785411" watchObservedRunningTime="2026-07-24 01:31:01.211574731 +0000 UTC m=+81.633641569"
Jul 24 01:31:02.190883 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:02.190361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft" event={"ID":"3e0f32d1-116e-47b7-be49-9928e551ca9e","Type":"ContainerStarted","Data":"c022ce714e99bd7a08440a14f24fadd054123e5652b2fbbfaf216f8152c1b365"}
Jul 24 01:31:02.208818 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:02.208750    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-rcnft" podStartSLOduration=39.669602129 podStartE2EDuration="41.208730124s" totalImagesPullingTime="1.539127995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:02.207450212 +0000 UTC m=+82.629517076" watchObservedRunningTime="2026-07-24 01:31:02.208730124 +0000 UTC m=+82.630796962"
Jul 24 01:31:04.199771 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:04.199735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2dpdk" event={"ID":"cb4d617d-edca-4b97-a24a-23449fe67b82","Type":"ContainerStarted","Data":"6fba921586ff5892704e074d7fb7c943b4980e10e4157493ac0e6d457f2b1f73"}
Jul 24 01:31:04.216724 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:04.216670    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2dpdk" podStartSLOduration=17.420214728 podStartE2EDuration="35.21665421s" totalImagesPullingTime="17.796439482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:30:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:04.214609353 +0000 UTC m=+84.636676188" watchObservedRunningTime="2026-07-24 01:31:04.21665421 +0000 UTC m=+84.638721045"
Jul 24 01:31:11.193126 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.193095    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z6ckz"
Jul 24 01:31:11.756078 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.756015    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"]
Jul 24 01:31:11.758838 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.758813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:11.761073 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.761051    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 01:31:11.761207 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.761151    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jxxbv\""
Jul 24 01:31:11.768010 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.767983    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"]
Jul 24 01:31:11.805896 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.805862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f444e058-9325-428d-9061-bee7ea74df28-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-sgwxx\" (UID: \"f444e058-9325-428d-9061-bee7ea74df28\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:11.907143 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.907104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f444e058-9325-428d-9061-bee7ea74df28-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-sgwxx\" (UID: \"f444e058-9325-428d-9061-bee7ea74df28\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:11.909568 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:11.909538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f444e058-9325-428d-9061-bee7ea74df28-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-sgwxx\" (UID: \"f444e058-9325-428d-9061-bee7ea74df28\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:12.068347 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:12.068268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:12.207787 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:31:12.207742    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf444e058_9325_428d_9061_bee7ea74df28.slice/crio-51c02a5a959b094edb1a2407d60001716a2c2933d038530fc5e3d19158bedc7e WatchSource:0}: Error finding container 51c02a5a959b094edb1a2407d60001716a2c2933d038530fc5e3d19158bedc7e: Status 404 returned error can't find the container with id 51c02a5a959b094edb1a2407d60001716a2c2933d038530fc5e3d19158bedc7e
Jul 24 01:31:12.209871 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:12.209823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"]
Jul 24 01:31:12.227974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:12.227941    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx" event={"ID":"f444e058-9325-428d-9061-bee7ea74df28","Type":"ContainerStarted","Data":"51c02a5a959b094edb1a2407d60001716a2c2933d038530fc5e3d19158bedc7e"}
Jul 24 01:31:13.133615 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:13.133586    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:31:13.930983 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:13.930944    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8577649d7d-4btcd"]
Jul 24 01:31:14.236533 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:14.236494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx" event={"ID":"f444e058-9325-428d-9061-bee7ea74df28","Type":"ContainerStarted","Data":"3c1c98a16a1c829f622fdb2a52cb1306d33f65882fff4a2cd7dcb0935416fa98"}
Jul 24 01:31:14.236714 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:14.236687    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:14.241404 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:14.241383    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx"
Jul 24 01:31:14.250840 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:14.250798    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-sgwxx" podStartSLOduration=2.05413624 podStartE2EDuration="3.250783399s" totalImagesPullingTime="1.196647159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:14.250062485 +0000 UTC m=+94.672129338" watchObservedRunningTime="2026-07-24 01:31:14.250783399 +0000 UTC m=+94.672850252"
Jul 24 01:31:20.145324 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.145285    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"]
Jul 24 01:31:20.148702 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.148674    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sh4bl"]
Jul 24 01:31:20.148834 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.148788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.151354 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.151334    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 01:31:20.151499 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.151481    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 01:31:20.152258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.152188    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 01:31:20.152258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.152219    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 01:31:20.152258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.152198    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 01:31:20.152498 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.152219    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 01:31:20.152498 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.152302    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.152498 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.152192    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-57khs\""
Jul 24 01:31:20.154266 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.154246    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 01:31:20.154563 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.154545    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 01:31:20.154622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.154575    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p46tp\""
Jul 24 01:31:20.154670 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.154575    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 01:31:20.165382 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.165342    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"]
Jul 24 01:31:20.270400 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270363    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9085d8f-b04c-4c20-80e8-c8201f767063-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.270400 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-textfile\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.270651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-accelerators-collector-config\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-sys\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-root\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.270860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9085d8f-b04c-4c20-80e8-c8201f767063-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.270860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.270860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-wtmp\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzwj\" (UniqueName: \"kubernetes.io/projected/35a3a0b8-4f5d-4580-b7bc-e6c241104637-kube-api-access-fqzwj\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.270860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270803    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9l7\" (UniqueName: \"kubernetes.io/projected/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-api-access-dm9l7\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.270860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270848    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a3a0b8-4f5d-4580-b7bc-e6c241104637-metrics-client-ca\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.271092 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.270892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.371505 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9l7\" (UniqueName: \"kubernetes.io/projected/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-api-access-dm9l7\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.371505 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a3a0b8-4f5d-4580-b7bc-e6c241104637-metrics-client-ca\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.371706 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.371706 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9085d8f-b04c-4c20-80e8-c8201f767063-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.371706 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.371839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-textfile\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.371839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.371839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-accelerators-collector-config\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.371839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-sys\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-root\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:20.371868    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:20.371936    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls podName:35a3a0b8-4f5d-4580-b7bc-e6c241104637 nodeName:}" failed. No retries permitted until 2026-07-24 01:31:20.871915256 +0000 UTC m=+101.293982087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls") pod "node-exporter-sh4bl" (UID: "35a3a0b8-4f5d-4580-b7bc-e6c241104637") : secret "node-exporter-tls" not found
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:20.371981    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.371988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9085d8f-b04c-4c20-80e8-c8201f767063-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.372046 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-sys\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372460 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-root\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372460 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372252    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a3a0b8-4f5d-4580-b7bc-e6c241104637-metrics-client-ca\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372460 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-textfile\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372460 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9085d8f-b04c-4c20-80e8-c8201f767063-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.372660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9085d8f-b04c-4c20-80e8-c8201f767063-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.372660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.372660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-accelerators-collector-config\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372660 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:20.372577    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-tls podName:d9085d8f-b04c-4c20-80e8-c8201f767063 nodeName:}" failed. No retries permitted until 2026-07-24 01:31:20.872555977 +0000 UTC m=+101.294622803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-xbqbq" (UID: "d9085d8f-b04c-4c20-80e8-c8201f767063") : secret "kube-state-metrics-tls" not found
Jul 24 01:31:20.372660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.372660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-wtmp\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372948 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzwj\" (UniqueName: \"kubernetes.io/projected/35a3a0b8-4f5d-4580-b7bc-e6c241104637-kube-api-access-fqzwj\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.372948 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.372786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-wtmp\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.374232 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.374211    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.374847 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.374822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.386634 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.386601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzwj\" (UniqueName: \"kubernetes.io/projected/35a3a0b8-4f5d-4580-b7bc-e6c241104637-kube-api-access-fqzwj\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.386634 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.386630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9l7\" (UniqueName: \"kubernetes.io/projected/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-api-access-dm9l7\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.877564 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.877521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:20.877765 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.877587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:20.877765 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:20.877682    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 01:31:20.877765 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:20.877748    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls podName:35a3a0b8-4f5d-4580-b7bc-e6c241104637 nodeName:}" failed. No retries permitted until 2026-07-24 01:31:21.877729274 +0000 UTC m=+102.299796089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls") pod "node-exporter-sh4bl" (UID: "35a3a0b8-4f5d-4580-b7bc-e6c241104637") : secret "node-exporter-tls" not found
Jul 24 01:31:20.879976 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:20.879945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9085d8f-b04c-4c20-80e8-c8201f767063-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xbqbq\" (UID: \"d9085d8f-b04c-4c20-80e8-c8201f767063\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:21.059775 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:21.059738    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"
Jul 24 01:31:21.197720 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:21.197696    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq"]
Jul 24 01:31:21.199753 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:31:21.199725    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9085d8f_b04c_4c20_80e8_c8201f767063.slice/crio-d1184f398e85d99bb12786b92d6ac61aac93a921cf149a078c9c5cea1fd703ef WatchSource:0}: Error finding container d1184f398e85d99bb12786b92d6ac61aac93a921cf149a078c9c5cea1fd703ef: Status 404 returned error can't find the container with id d1184f398e85d99bb12786b92d6ac61aac93a921cf149a078c9c5cea1fd703ef
Jul 24 01:31:21.261437 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:21.261403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq" event={"ID":"d9085d8f-b04c-4c20-80e8-c8201f767063","Type":"ContainerStarted","Data":"d1184f398e85d99bb12786b92d6ac61aac93a921cf149a078c9c5cea1fd703ef"}
Jul 24 01:31:21.886257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:21.886217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:21.888537 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:21.888516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35a3a0b8-4f5d-4580-b7bc-e6c241104637-node-exporter-tls\") pod \"node-exporter-sh4bl\" (UID: \"35a3a0b8-4f5d-4580-b7bc-e6c241104637\") " pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:21.965435 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:21.965388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sh4bl"
Jul 24 01:31:21.977602 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:31:21.977573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a3a0b8_4f5d_4580_b7bc_e6c241104637.slice/crio-ad3d9c40c489d40f3e8edfe6a20a23e6da2581c72d42dfb2695357ba1ecd472d WatchSource:0}: Error finding container ad3d9c40c489d40f3e8edfe6a20a23e6da2581c72d42dfb2695357ba1ecd472d: Status 404 returned error can't find the container with id ad3d9c40c489d40f3e8edfe6a20a23e6da2581c72d42dfb2695357ba1ecd472d
Jul 24 01:31:22.109297 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.109265    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-964f86565-phch7"]
Jul 24 01:31:22.113989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.113964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.116757    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3m9e2mkpr31jr\""
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.117066    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.117250    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.117448    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.117583    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.117680    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hqmcr\""
Jul 24 01:31:22.118837 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.118637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 01:31:22.124451 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.124427    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-964f86565-phch7"]
Jul 24 01:31:22.188466 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188466 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-grpc-tls\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-metrics-client-ca\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-tls\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.188657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.188632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62zn\" (UniqueName: \"kubernetes.io/projected/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-kube-api-access-c62zn\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.266907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.266863    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh4bl" event={"ID":"35a3a0b8-4f5d-4580-b7bc-e6c241104637","Type":"ContainerStarted","Data":"ad3d9c40c489d40f3e8edfe6a20a23e6da2581c72d42dfb2695357ba1ecd472d"}
Jul 24 01:31:22.289126 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289310 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-grpc-tls\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289310 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289310 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-metrics-client-ca\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289310 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-tls\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289310 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289532 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.289532 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.289446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62zn\" (UniqueName: \"kubernetes.io/projected/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-kube-api-access-c62zn\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.290832 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.290776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-metrics-client-ca\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.292958 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.292894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-grpc-tls\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.292958 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.292909    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.293180 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.293000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.293180 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.293147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.293538 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.293512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-tls\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.293936 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.293914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.297343 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.297317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62zn\" (UniqueName: \"kubernetes.io/projected/ba3ca594-7a95-4965-a2c9-ae5b9e09794d-kube-api-access-c62zn\") pod \"thanos-querier-964f86565-phch7\" (UID: \"ba3ca594-7a95-4965-a2c9-ae5b9e09794d\") " pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.429007 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.428909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:22.662277 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:22.662251    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-964f86565-phch7"]
Jul 24 01:31:22.667128 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:31:22.667093    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3ca594_7a95_4965_a2c9_ae5b9e09794d.slice/crio-0a0029583be2b3e112f8928178aaa789c936899661b7f86f0d5654f28f1bb992 WatchSource:0}: Error finding container 0a0029583be2b3e112f8928178aaa789c936899661b7f86f0d5654f28f1bb992: Status 404 returned error can't find the container with id 0a0029583be2b3e112f8928178aaa789c936899661b7f86f0d5654f28f1bb992
Jul 24 01:31:23.271638 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:23.271606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"0a0029583be2b3e112f8928178aaa789c936899661b7f86f0d5654f28f1bb992"}
Jul 24 01:31:23.273693 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:23.273669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq" event={"ID":"d9085d8f-b04c-4c20-80e8-c8201f767063","Type":"ContainerStarted","Data":"abdd66743bd1c228f5ca17674f7ba1c79329f3b5e555715e8da66aa4945467fa"}
Jul 24 01:31:23.273775 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:23.273719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq" event={"ID":"d9085d8f-b04c-4c20-80e8-c8201f767063","Type":"ContainerStarted","Data":"efe25dfbc4ac31982e792f3c38194dd32ca13ecb4981e52a9e8ef780b99bb71d"}
Jul 24 01:31:23.273775 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:23.273740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq" event={"ID":"d9085d8f-b04c-4c20-80e8-c8201f767063","Type":"ContainerStarted","Data":"3a86d6899f5b3386e0c878f47deda9c269c65a48a0f6c8244c77abc0f5bf4cb4"}
Jul 24 01:31:23.310358 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:23.310288    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xbqbq" podStartSLOduration=1.95783919 podStartE2EDuration="3.310267352s" totalImagesPullingTime="1.352428162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:23.308263126 +0000 UTC m=+103.730329983" watchObservedRunningTime="2026-07-24 01:31:23.310267352 +0000 UTC m=+103.732334189"
Jul 24 01:31:24.136637 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.136603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9kkdx"
Jul 24 01:31:24.280858 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.280819    2564 generic.go:356] "Generic (PLEG): container finished" podID="35a3a0b8-4f5d-4580-b7bc-e6c241104637" containerID="6d3cf8639d446bb8312bbbbc9e6b929f9879873a603c6a321bb2ac78ba28e615" exitCode=0
Jul 24 01:31:24.281359 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.280922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh4bl" event={"ID":"35a3a0b8-4f5d-4580-b7bc-e6c241104637","Type":"ContainerDied","Data":"6d3cf8639d446bb8312bbbbc9e6b929f9879873a603c6a321bb2ac78ba28e615"}
Jul 24 01:31:24.535722 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.535677    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f86786855-m5886"]
Jul 24 01:31:24.555294 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.555256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f86786855-m5886"]
Jul 24 01:31:24.555469 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.555298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.558058 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.557826    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 01:31:24.558058 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.557862    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 01:31:24.558058 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.557887    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tnh2q\""
Jul 24 01:31:24.558058 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.557916    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 01:31:24.558058 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.557890    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 01:31:24.558058 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.558044    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-anlgtbe0bjpbe\""
Jul 24 01:31:24.610045 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.609992    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-secret-metrics-server-tls\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.610210 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.610066    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-client-ca-bundle\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.610210 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.610099    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d326a642-01d8-4477-bc8c-d6d918ff8130-metrics-server-audit-profiles\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.610341 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.610207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d326a642-01d8-4477-bc8c-d6d918ff8130-audit-log\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.610341 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.610251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-secret-metrics-server-client-certs\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.610341 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.610291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d326a642-01d8-4477-bc8c-d6d918ff8130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.610458 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.610375    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkffv\" (UniqueName: \"kubernetes.io/projected/d326a642-01d8-4477-bc8c-d6d918ff8130-kube-api-access-rkffv\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.711613 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d326a642-01d8-4477-bc8c-d6d918ff8130-audit-log\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.711798 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-secret-metrics-server-client-certs\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.711798 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d326a642-01d8-4477-bc8c-d6d918ff8130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.711798 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkffv\" (UniqueName: \"kubernetes.io/projected/d326a642-01d8-4477-bc8c-d6d918ff8130-kube-api-access-rkffv\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.711798 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-secret-metrics-server-tls\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.711798 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-client-ca-bundle\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.712079 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.711799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d326a642-01d8-4477-bc8c-d6d918ff8130-metrics-server-audit-profiles\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.712079 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.712003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d326a642-01d8-4477-bc8c-d6d918ff8130-audit-log\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.712469 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.712441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d326a642-01d8-4477-bc8c-d6d918ff8130-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.712914 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.712890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d326a642-01d8-4477-bc8c-d6d918ff8130-metrics-server-audit-profiles\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.714781 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.714758    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-secret-metrics-server-client-certs\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.714878 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.714817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-secret-metrics-server-tls\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.714946 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.714897    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d326a642-01d8-4477-bc8c-d6d918ff8130-client-ca-bundle\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.719743 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.719721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkffv\" (UniqueName: \"kubernetes.io/projected/d326a642-01d8-4477-bc8c-d6d918ff8130-kube-api-access-rkffv\") pod \"metrics-server-7f86786855-m5886\" (UID: \"d326a642-01d8-4477-bc8c-d6d918ff8130\") " pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:24.867488 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:24.867145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:25.089854 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:25.089800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f86786855-m5886"]
Jul 24 01:31:25.093041 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:31:25.092991    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd326a642_01d8_4477_bc8c_d6d918ff8130.slice/crio-8ac575c69de4c2e424d46b2c52552728d67a71c629cbb2efe71b04a3dd369a93 WatchSource:0}: Error finding container 8ac575c69de4c2e424d46b2c52552728d67a71c629cbb2efe71b04a3dd369a93: Status 404 returned error can't find the container with id 8ac575c69de4c2e424d46b2c52552728d67a71c629cbb2efe71b04a3dd369a93
Jul 24 01:31:25.289831 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:25.289743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh4bl" event={"ID":"35a3a0b8-4f5d-4580-b7bc-e6c241104637","Type":"ContainerStarted","Data":"aa797fa4c0e1958ed35f319f6a9038b8f879357f678088d18d5e0b86676e84d1"}
Jul 24 01:31:25.289831 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:25.289806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sh4bl" event={"ID":"35a3a0b8-4f5d-4580-b7bc-e6c241104637","Type":"ContainerStarted","Data":"4d9c7eb4cc4073c0d428a3d4e501f1a0f9fe70b0b5cb08ebea16c427ae770843"}
Jul 24 01:31:25.291062 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:25.290976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f86786855-m5886" event={"ID":"d326a642-01d8-4477-bc8c-d6d918ff8130","Type":"ContainerStarted","Data":"8ac575c69de4c2e424d46b2c52552728d67a71c629cbb2efe71b04a3dd369a93"}
Jul 24 01:31:25.306092 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:25.306021    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sh4bl" podStartSLOduration=3.95013856 podStartE2EDuration="5.306000961s" totalImagesPullingTime="1.281182173s" totalInitContainerRuntime="74.680228ms" isStatefulPod=true podCreationTimestamp="2026-07-24 01:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:25.305472725 +0000 UTC m=+105.727539601" watchObservedRunningTime="2026-07-24 01:31:25.306000961 +0000 UTC m=+105.728067797"
Jul 24 01:31:26.296981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:26.296946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"8b42f9f24e48f17fdc59f7d4873234304a406ea064059356106f0762d1b9b48e"}
Jul 24 01:31:26.297365 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:26.297021    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"bd5c9beb94e7c2b840b34151b8cf6f3ab4edbc5836ba5f43b33c3eadbd8b7145"}
Jul 24 01:31:26.297365 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:26.297065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"636afc7a0f0dac88aebe637edc2908aab991ff5a53975b457b6a3bccd551081e"}
Jul 24 01:31:27.304504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:27.304414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"7fee9b7aae2fc993251ba0371ba40a210531d34069b3be63afac92febccac737"}
Jul 24 01:31:28.310114 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:28.310074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f86786855-m5886" event={"ID":"d326a642-01d8-4477-bc8c-d6d918ff8130","Type":"ContainerStarted","Data":"0227a1d784f80d56586f206b8908f1bb9be7d9a1d78f5f75c93c28e80e7e73c1"}
Jul 24 01:31:28.313217 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:28.313184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"1bc9773e4f601819732642757f50e002d836684dc103c755f9691673fc5c3684"}
Jul 24 01:31:28.313362 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:28.313255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-964f86565-phch7" event={"ID":"ba3ca594-7a95-4965-a2c9-ae5b9e09794d","Type":"ContainerStarted","Data":"a5407835c1bf423edcb6346bc0b3fd23b2264e9baf36dd566a162c262de6e9fa"}
Jul 24 01:31:28.313362 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:28.313317    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:28.330145 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:28.330099    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f86786855-m5886" podStartSLOduration=1.552057437 podStartE2EDuration="4.330083971s" totalImagesPullingTime="2.778026534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:28.328596894 +0000 UTC m=+108.750663776" watchObservedRunningTime="2026-07-24 01:31:28.330083971 +0000 UTC m=+108.752150861"
Jul 24 01:31:28.349547 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:28.349483    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-964f86565-phch7" podStartSLOduration=2.322457951 podStartE2EDuration="6.349455659s" totalImagesPullingTime="4.026997708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:31:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:31:28.349371956 +0000 UTC m=+108.771438813" watchObservedRunningTime="2026-07-24 01:31:28.349455659 +0000 UTC m=+108.771522496"
Jul 24 01:31:34.326202 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:34.326172    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-964f86565-phch7"
Jul 24 01:31:38.950214 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:38.950169    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8577649d7d-4btcd" podUID="ae8f6899-df6c-445c-98b4-856f9af10e14" containerName="registry" containerID="cri-o://27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84" gracePeriod=30
Jul 24 01:31:39.186401 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.186361    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:31:39.330956 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.330869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-image-registry-private-configuration\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.330956 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.330925    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae8f6899-df6c-445c-98b4-856f9af10e14-ca-trust-extracted\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.330956 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.330947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-bound-sa-token\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.331259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-certificates\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.331259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.331259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331226    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-installation-pull-secrets\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.331259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331253    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-kube-api-access-nzk7l\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.331450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331278    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-trusted-ca\") pod \"ae8f6899-df6c-445c-98b4-856f9af10e14\" (UID: \"ae8f6899-df6c-445c-98b4-856f9af10e14\") "
Jul 24 01:31:39.331552 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331510    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:31:39.331947 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.331908    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:31:39.333819 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.333786    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:31:39.333819 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.333812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:31:39.334084 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.333968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:31:39.334084 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.333996    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:31:39.334243 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.334224    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-kube-api-access-nzk7l" (OuterVolumeSpecName: "kube-api-access-nzk7l") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "kube-api-access-nzk7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:31:39.339755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.339733    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8f6899-df6c-445c-98b4-856f9af10e14-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ae8f6899-df6c-445c-98b4-856f9af10e14" (UID: "ae8f6899-df6c-445c-98b4-856f9af10e14"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:31:39.359786 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.359763    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a755877-b146-4230-8b54-d66daa557182" containerID="ae42a568194617b631c66ebc337394d34a32490e443a27ae6ff23d3167490d79" exitCode=0
Jul 24 01:31:39.359892 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.359841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465" event={"ID":"7a755877-b146-4230-8b54-d66daa557182","Type":"ContainerDied","Data":"ae42a568194617b631c66ebc337394d34a32490e443a27ae6ff23d3167490d79"}
Jul 24 01:31:39.360242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.360226    2564 scope.go:117] "RemoveContainer" containerID="ae42a568194617b631c66ebc337394d34a32490e443a27ae6ff23d3167490d79"
Jul 24 01:31:39.361148 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.361128    2564 generic.go:356] "Generic (PLEG): container finished" podID="ae8f6899-df6c-445c-98b4-856f9af10e14" containerID="27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84" exitCode=0
Jul 24 01:31:39.361245 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.361158    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8577649d7d-4btcd" event={"ID":"ae8f6899-df6c-445c-98b4-856f9af10e14","Type":"ContainerDied","Data":"27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84"}
Jul 24 01:31:39.361307 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.361253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8577649d7d-4btcd" event={"ID":"ae8f6899-df6c-445c-98b4-856f9af10e14","Type":"ContainerDied","Data":"baec91a63f93fc792b2be2917fc406764ea02f542f89b88ec92b207d7500ef5c"}
Jul 24 01:31:39.361307 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.361181    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8577649d7d-4btcd"
Jul 24 01:31:39.361307 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.361283    2564 scope.go:117] "RemoveContainer" containerID="27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84"
Jul 24 01:31:39.370400 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.370382    2564 scope.go:117] "RemoveContainer" containerID="27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84"
Jul 24 01:31:39.370696 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:31:39.370664    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84\": container with ID starting with 27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84 not found: ID does not exist" containerID="27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84"
Jul 24 01:31:39.370770 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.370708    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84"} err="failed to get container status \"27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84\": rpc error: code = NotFound desc = could not find container \"27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84\": container with ID starting with 27ba56f1a5be719fa0765a485572864aff2217b78481bd9adb97dda43adb7e84 not found: ID does not exist"
Jul 24 01:31:39.390961 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.390901    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8577649d7d-4btcd"]
Jul 24 01:31:39.401184 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.401156    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8577649d7d-4btcd"]
Jul 24 01:31:39.432257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432220    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-certificates\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432248    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-registry-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432262    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-installation-pull-secrets\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432465 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432275    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-kube-api-access-nzk7l\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432465 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432290    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f6899-df6c-445c-98b4-856f9af10e14-trusted-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432465 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432303    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae8f6899-df6c-445c-98b4-856f9af10e14-image-registry-private-configuration\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432465 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432315    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae8f6899-df6c-445c-98b4-856f9af10e14-ca-trust-extracted\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:39.432465 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:39.432328    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae8f6899-df6c-445c-98b4-856f9af10e14-bound-sa-token\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:31:40.209437 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:40.209404    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8f6899-df6c-445c-98b4-856f9af10e14" path="/var/lib/kubelet/pods/ae8f6899-df6c-445c-98b4-856f9af10e14/volumes"
Jul 24 01:31:40.366384 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:40.366302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-lq465" event={"ID":"7a755877-b146-4230-8b54-d66daa557182","Type":"ContainerStarted","Data":"d99cfba4086f8922e3c6363f9ec8a3da5e5df667a5300759760df555b010fac2"}
Jul 24 01:31:44.384320 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:44.384282    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f35de79-075f-4d52-b979-c0d51c69722d" containerID="2697351412c2cd2497499dd59b0b5ca3b296523a6701fb2298ade406a10bbc7b" exitCode=0
Jul 24 01:31:44.384750 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:44.384353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-4rr9p" event={"ID":"1f35de79-075f-4d52-b979-c0d51c69722d","Type":"ContainerDied","Data":"2697351412c2cd2497499dd59b0b5ca3b296523a6701fb2298ade406a10bbc7b"}
Jul 24 01:31:44.384750 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:44.384696    2564 scope.go:117] "RemoveContainer" containerID="2697351412c2cd2497499dd59b0b5ca3b296523a6701fb2298ade406a10bbc7b"
Jul 24 01:31:44.867692 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:44.867652    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:44.867868 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:44.867721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:31:45.390120 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:31:45.390079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-4rr9p" event={"ID":"1f35de79-075f-4d52-b979-c0d51c69722d","Type":"ContainerStarted","Data":"91028218a12971f17f744f396c53b036b9b8a333a15c99e59c927b5d62140022"}
Jul 24 01:32:04.873549 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:04.873516    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:32:04.877490 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:04.877465    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f86786855-m5886"
Jul 24 01:32:44.466529 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.466479    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"]
Jul 24 01:32:44.466989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.466870    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8f6899-df6c-445c-98b4-856f9af10e14" containerName="registry"
Jul 24 01:32:44.466989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.466887    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8f6899-df6c-445c-98b4-856f9af10e14" containerName="registry"
Jul 24 01:32:44.466989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.466935    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8f6899-df6c-445c-98b4-856f9af10e14" containerName="registry"
Jul 24 01:32:44.469849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.469831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.473915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.472863    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 01:32:44.473915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.473240    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 01:32:44.473915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.473541    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 01:32:44.473915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.473589    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zn22m\""
Jul 24 01:32:44.474502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.474483    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 01:32:44.477574 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.477548    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 01:32:44.480654 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.480632    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 01:32:44.481668 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.481326    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"]
Jul 24 01:32:44.537677 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-secret-telemeter-client\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-metrics-client-ca\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537717    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-federate-client-tls\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-serving-certs-ca-bundle\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2gq\" (UniqueName: \"kubernetes.io/projected/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-kube-api-access-nv2gq\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-telemeter-client-tls\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.537849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.537848    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639224 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-metrics-client-ca\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639224 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-federate-client-tls\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-serving-certs-ca-bundle\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2gq\" (UniqueName: \"kubernetes.io/projected/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-kube-api-access-nv2gq\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-telemeter-client-tls\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.639489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.639429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-secret-telemeter-client\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.640171 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.640143    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-metrics-client-ca\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.640351 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.640165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-serving-certs-ca-bundle\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.640588 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.640568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.641981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.641956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-secret-telemeter-client\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.642133 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.642116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-telemeter-client-tls\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.642213 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.642197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.642399 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.642379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-federate-client-tls\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.647646 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.647624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2gq\" (UniqueName: \"kubernetes.io/projected/87ad5f7b-e7aa-4b26-95e5-f4d1959e8484-kube-api-access-nv2gq\") pod \"telemeter-client-75f444cc7f-wb5nt\" (UID: \"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484\") " pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.785137 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.785056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"
Jul 24 01:32:44.920334 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:44.920308    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75f444cc7f-wb5nt"]
Jul 24 01:32:44.922803 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:32:44.922776    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ad5f7b_e7aa_4b26_95e5_f4d1959e8484.slice/crio-cc522052eb3f404d7480ca111d777dafbd2e016f81a46e939d7179dfcfb8813a WatchSource:0}: Error finding container cc522052eb3f404d7480ca111d777dafbd2e016f81a46e939d7179dfcfb8813a: Status 404 returned error can't find the container with id cc522052eb3f404d7480ca111d777dafbd2e016f81a46e939d7179dfcfb8813a
Jul 24 01:32:45.617356 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:45.617323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt" event={"ID":"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484","Type":"ContainerStarted","Data":"cc522052eb3f404d7480ca111d777dafbd2e016f81a46e939d7179dfcfb8813a"}
Jul 24 01:32:47.629881 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:47.629843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt" event={"ID":"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484","Type":"ContainerStarted","Data":"1313aeec86d4d6699d8979396bcc775f02434ed022334303a75b3b19a4695f0c"}
Jul 24 01:32:48.635666 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:48.635582    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt" event={"ID":"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484","Type":"ContainerStarted","Data":"ceb36f43c3b28043c86a900d67ecc0813b145db123a5dca38acca384d4d9257a"}
Jul 24 01:32:48.635666 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:48.635641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt" event={"ID":"87ad5f7b-e7aa-4b26-95e5-f4d1959e8484","Type":"ContainerStarted","Data":"3c9620f4a9e8f8d89e95ca7fa9a69f6cd2c356197a703902076fc70e94077740"}
Jul 24 01:32:48.655396 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:32:48.655338    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75f444cc7f-wb5nt" podStartSLOduration=1.434588892 podStartE2EDuration="4.655323163s" totalImagesPullingTime="3.220734271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:32:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:32:48.655018309 +0000 UTC m=+189.077085154" watchObservedRunningTime="2026-07-24 01:32:48.655323163 +0000 UTC m=+189.077389999"
Jul 24 01:34:02.993875 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:02.993795    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"]
Jul 24 01:34:02.997155 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:02.997133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:02.999922 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:02.999901    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 01:34:03.000054 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.000016    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 01:34:03.000159 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.000140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5sgzw\""
Jul 24 01:34:03.005229 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.005208    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"]
Jul 24 01:34:03.175876 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.175817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.176093 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.175926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.176093 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.175984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dc44\" (UniqueName: \"kubernetes.io/projected/7c2f0d05-050c-4222-9cf3-c88a17c5030e-kube-api-access-6dc44\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.276461 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.276376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.276461 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.276422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dc44\" (UniqueName: \"kubernetes.io/projected/7c2f0d05-050c-4222-9cf3-c88a17c5030e-kube-api-access-6dc44\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.276627 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.276470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.276827 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.276803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.276867 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.276810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.284575 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.284545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dc44\" (UniqueName: \"kubernetes.io/projected/7c2f0d05-050c-4222-9cf3-c88a17c5030e-kube-api-access-6dc44\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.306490 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.306465    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:03.435481 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.435446    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"]
Jul 24 01:34:03.439273 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:03.439244    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2f0d05_050c_4222_9cf3_c88a17c5030e.slice/crio-890f7e7867dd84228ebd91d76cb277c334df5d0519366e5cb6cacab33ef90502 WatchSource:0}: Error finding container 890f7e7867dd84228ebd91d76cb277c334df5d0519366e5cb6cacab33ef90502: Status 404 returned error can't find the container with id 890f7e7867dd84228ebd91d76cb277c334df5d0519366e5cb6cacab33ef90502
Jul 24 01:34:03.932329 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:03.932289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerStarted","Data":"890f7e7867dd84228ebd91d76cb277c334df5d0519366e5cb6cacab33ef90502"}
Jul 24 01:34:09.961159 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:09.961114    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerStarted","Data":"3902c4f6d234c1a6070e4c651e7f3ace7dc2f5a6838464b348aaa73b389ce2a5"}
Jul 24 01:34:10.966522 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:10.966489    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerID="3902c4f6d234c1a6070e4c651e7f3ace7dc2f5a6838464b348aaa73b389ce2a5" exitCode=0
Jul 24 01:34:10.966876 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:10.966528    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerDied","Data":"3902c4f6d234c1a6070e4c651e7f3ace7dc2f5a6838464b348aaa73b389ce2a5"}
Jul 24 01:34:12.977827 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:12.977796    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerStarted","Data":"be57ef74e110356076d412df611adaf69d654e1b6960e5e49555131c1992a97e"}
Jul 24 01:34:13.983170 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:13.983134    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerID="be57ef74e110356076d412df611adaf69d654e1b6960e5e49555131c1992a97e" exitCode=0
Jul 24 01:34:13.983555 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:13.983220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerDied","Data":"be57ef74e110356076d412df611adaf69d654e1b6960e5e49555131c1992a97e"}
Jul 24 01:34:21.015623 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:21.015581    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerID="d2524f9e7a60a70ae7e107686d677649556b2ee7802977dab2043583d27f8e9e" exitCode=0
Jul 24 01:34:21.015993 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:21.015651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerDied","Data":"d2524f9e7a60a70ae7e107686d677649556b2ee7802977dab2043583d27f8e9e"}
Jul 24 01:34:22.153738 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.153708    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:22.253384 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.253342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-bundle\") pod \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") "
Jul 24 01:34:22.253577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.253417    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-util\") pod \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") "
Jul 24 01:34:22.253577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.253492    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dc44\" (UniqueName: \"kubernetes.io/projected/7c2f0d05-050c-4222-9cf3-c88a17c5030e-kube-api-access-6dc44\") pod \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\" (UID: \"7c2f0d05-050c-4222-9cf3-c88a17c5030e\") "
Jul 24 01:34:22.253971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.253943    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-bundle" (OuterVolumeSpecName: "bundle") pod "7c2f0d05-050c-4222-9cf3-c88a17c5030e" (UID: "7c2f0d05-050c-4222-9cf3-c88a17c5030e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:34:22.255715 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.255683    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2f0d05-050c-4222-9cf3-c88a17c5030e-kube-api-access-6dc44" (OuterVolumeSpecName: "kube-api-access-6dc44") pod "7c2f0d05-050c-4222-9cf3-c88a17c5030e" (UID: "7c2f0d05-050c-4222-9cf3-c88a17c5030e"). InnerVolumeSpecName "kube-api-access-6dc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:34:22.257987 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.257967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-util" (OuterVolumeSpecName: "util") pod "7c2f0d05-050c-4222-9cf3-c88a17c5030e" (UID: "7c2f0d05-050c-4222-9cf3-c88a17c5030e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:34:22.355002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.354922    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:22.355002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.354951    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dc44\" (UniqueName: \"kubernetes.io/projected/7c2f0d05-050c-4222-9cf3-c88a17c5030e-kube-api-access-6dc44\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:22.355002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:22.354963    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c2f0d05-050c-4222-9cf3-c88a17c5030e-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:23.024999 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:23.024963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9" event={"ID":"7c2f0d05-050c-4222-9cf3-c88a17c5030e","Type":"ContainerDied","Data":"890f7e7867dd84228ebd91d76cb277c334df5d0519366e5cb6cacab33ef90502"}
Jul 24 01:34:23.024999 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:23.024997    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540prqd9"
Jul 24 01:34:23.025285 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:23.025045    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890f7e7867dd84228ebd91d76cb277c334df5d0519366e5cb6cacab33ef90502"
Jul 24 01:34:30.752592 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752556    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"]
Jul 24 01:34:30.752957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752882    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="util"
Jul 24 01:34:30.752957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752893    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="util"
Jul 24 01:34:30.752957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752901    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="extract"
Jul 24 01:34:30.752957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752906    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="extract"
Jul 24 01:34:30.752957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752920    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="pull"
Jul 24 01:34:30.752957 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752927    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="pull"
Jul 24 01:34:30.753162 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.752980    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2f0d05-050c-4222-9cf3-c88a17c5030e" containerName="extract"
Jul 24 01:34:30.756424 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.756405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:30.759941 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.759920    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kqwmp\""
Jul 24 01:34:30.760038 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.759984    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 24 01:34:30.760088 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.760054    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 24 01:34:30.782315 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.782281    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"]
Jul 24 01:34:30.824664 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.824629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5s9w\" (UniqueName: \"kubernetes.io/projected/f89bb5fd-7e74-4307-80f6-0a96fe312325-kube-api-access-s5s9w\") pod \"cert-manager-operator-controller-manager-55c444bf86-2lj5c\" (UID: \"f89bb5fd-7e74-4307-80f6-0a96fe312325\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:30.824805 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.824669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89bb5fd-7e74-4307-80f6-0a96fe312325-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2lj5c\" (UID: \"f89bb5fd-7e74-4307-80f6-0a96fe312325\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:30.925714 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.925681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5s9w\" (UniqueName: \"kubernetes.io/projected/f89bb5fd-7e74-4307-80f6-0a96fe312325-kube-api-access-s5s9w\") pod \"cert-manager-operator-controller-manager-55c444bf86-2lj5c\" (UID: \"f89bb5fd-7e74-4307-80f6-0a96fe312325\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:30.925714 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.925718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89bb5fd-7e74-4307-80f6-0a96fe312325-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2lj5c\" (UID: \"f89bb5fd-7e74-4307-80f6-0a96fe312325\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:30.926101 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.926079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89bb5fd-7e74-4307-80f6-0a96fe312325-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2lj5c\" (UID: \"f89bb5fd-7e74-4307-80f6-0a96fe312325\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:30.935711 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:30.935687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5s9w\" (UniqueName: \"kubernetes.io/projected/f89bb5fd-7e74-4307-80f6-0a96fe312325-kube-api-access-s5s9w\") pod \"cert-manager-operator-controller-manager-55c444bf86-2lj5c\" (UID: \"f89bb5fd-7e74-4307-80f6-0a96fe312325\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:31.065779 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:31.065702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"
Jul 24 01:34:31.203285 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:31.203251    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c"]
Jul 24 01:34:31.206951 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:31.206926    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89bb5fd_7e74_4307_80f6_0a96fe312325.slice/crio-e11a9faa4f95fe1e75bd0417b9974e432392ed5eedd82534605ab67a8323f7b2 WatchSource:0}: Error finding container e11a9faa4f95fe1e75bd0417b9974e432392ed5eedd82534605ab67a8323f7b2: Status 404 returned error can't find the container with id e11a9faa4f95fe1e75bd0417b9974e432392ed5eedd82534605ab67a8323f7b2
Jul 24 01:34:32.069850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:32.069806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c" event={"ID":"f89bb5fd-7e74-4307-80f6-0a96fe312325","Type":"ContainerStarted","Data":"e11a9faa4f95fe1e75bd0417b9974e432392ed5eedd82534605ab67a8323f7b2"}
Jul 24 01:34:33.076566 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:33.076530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c" event={"ID":"f89bb5fd-7e74-4307-80f6-0a96fe312325","Type":"ContainerStarted","Data":"eecaa3a303a6c4c02957eb5e97526f37df55466a75b7fa9b051726f5a65f5858"}
Jul 24 01:34:33.100241 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:33.100188    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2lj5c" podStartSLOduration=1.433976529 podStartE2EDuration="3.100173135s" totalImagesPullingTime="1.666196606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:34:33.099182263 +0000 UTC m=+293.521249101" watchObservedRunningTime="2026-07-24 01:34:33.100173135 +0000 UTC m=+293.522239971"
Jul 24 01:34:34.441719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.441669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"]
Jul 24 01:34:34.445205 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.445186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.447539 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.447514    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 01:34:34.448337 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.448315    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5sgzw\""
Jul 24 01:34:34.448438 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.448315    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 01:34:34.454481 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.454458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"]
Jul 24 01:34:34.555888 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.555838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjvn\" (UniqueName: \"kubernetes.io/projected/308cb2bd-bf7b-471b-a122-14899b8621da-kube-api-access-dxjvn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.556059 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.555909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.556059 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.555982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.656735 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.656699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.656906 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.656741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.656906 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.656782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjvn\" (UniqueName: \"kubernetes.io/projected/308cb2bd-bf7b-471b-a122-14899b8621da-kube-api-access-dxjvn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.657112 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.657090    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.657203 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.657179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.664252 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.664229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjvn\" (UniqueName: \"kubernetes.io/projected/308cb2bd-bf7b-471b-a122-14899b8621da-kube-api-access-dxjvn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.755222 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.755188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:34.886623 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:34.886590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"]
Jul 24 01:34:34.888963 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:34.888932    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308cb2bd_bf7b_471b_a122_14899b8621da.slice/crio-04affb399e92768e52284fd4994f0a182096b47dc75ab676cf5ba3f12115ac71 WatchSource:0}: Error finding container 04affb399e92768e52284fd4994f0a182096b47dc75ab676cf5ba3f12115ac71: Status 404 returned error can't find the container with id 04affb399e92768e52284fd4994f0a182096b47dc75ab676cf5ba3f12115ac71
Jul 24 01:34:35.086072 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:35.085959    2564 generic.go:356] "Generic (PLEG): container finished" podID="308cb2bd-bf7b-471b-a122-14899b8621da" containerID="0cc6642807e7cdab87341f437203cfef984c7f7a840ff4c7c689f3d1df79ca4f" exitCode=0
Jul 24 01:34:35.086072 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:35.086008    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4" event={"ID":"308cb2bd-bf7b-471b-a122-14899b8621da","Type":"ContainerDied","Data":"0cc6642807e7cdab87341f437203cfef984c7f7a840ff4c7c689f3d1df79ca4f"}
Jul 24 01:34:35.086259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:35.086087    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4" event={"ID":"308cb2bd-bf7b-471b-a122-14899b8621da","Type":"ContainerStarted","Data":"04affb399e92768e52284fd4994f0a182096b47dc75ab676cf5ba3f12115ac71"}
Jul 24 01:34:36.841495 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.841463    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f7hrn"]
Jul 24 01:34:36.850827 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.850802    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:36.852056 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.851979    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f7hrn"]
Jul 24 01:34:36.853321 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.853297    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 01:34:36.854304 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.854260    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g595s\""
Jul 24 01:34:36.854408 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.854326    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 01:34:36.976337 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.976304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35f10a4f-8e54-4c5f-b793-ce03f9247ca7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f7hrn\" (UID: \"35f10a4f-8e54-4c5f-b793-ce03f9247ca7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:36.976517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:36.976381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gm6\" (UniqueName: \"kubernetes.io/projected/35f10a4f-8e54-4c5f-b793-ce03f9247ca7-kube-api-access-r9gm6\") pod \"cert-manager-webhook-6486ffd997-f7hrn\" (UID: \"35f10a4f-8e54-4c5f-b793-ce03f9247ca7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:37.077518 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.077480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gm6\" (UniqueName: \"kubernetes.io/projected/35f10a4f-8e54-4c5f-b793-ce03f9247ca7-kube-api-access-r9gm6\") pod \"cert-manager-webhook-6486ffd997-f7hrn\" (UID: \"35f10a4f-8e54-4c5f-b793-ce03f9247ca7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:37.077680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.077584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35f10a4f-8e54-4c5f-b793-ce03f9247ca7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f7hrn\" (UID: \"35f10a4f-8e54-4c5f-b793-ce03f9247ca7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:37.085978 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.085949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35f10a4f-8e54-4c5f-b793-ce03f9247ca7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f7hrn\" (UID: \"35f10a4f-8e54-4c5f-b793-ce03f9247ca7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:37.086108 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.085981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gm6\" (UniqueName: \"kubernetes.io/projected/35f10a4f-8e54-4c5f-b793-ce03f9247ca7-kube-api-access-r9gm6\") pod \"cert-manager-webhook-6486ffd997-f7hrn\" (UID: \"35f10a4f-8e54-4c5f-b793-ce03f9247ca7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:37.171290 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.171207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:37.346756 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.346723    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f7hrn"]
Jul 24 01:34:37.349411 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:37.349385    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f10a4f_8e54_4c5f_b793_ce03f9247ca7.slice/crio-68f5f23f66a97d37b39cd6de6d4d72b799ae209f0e793df35756f4f65ce958b6 WatchSource:0}: Error finding container 68f5f23f66a97d37b39cd6de6d4d72b799ae209f0e793df35756f4f65ce958b6: Status 404 returned error can't find the container with id 68f5f23f66a97d37b39cd6de6d4d72b799ae209f0e793df35756f4f65ce958b6
Jul 24 01:34:37.436738 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.436657    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"]
Jul 24 01:34:37.440199 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.440181    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.442271 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.442248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-g4rdc\""
Jul 24 01:34:37.446486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.446466    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"]
Jul 24 01:34:37.581089 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.581047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f81bc1c6-4133-442a-a6f2-c83502b51f0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6nlsj\" (UID: \"f81bc1c6-4133-442a-a6f2-c83502b51f0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.581267 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.581149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xf9\" (UniqueName: \"kubernetes.io/projected/f81bc1c6-4133-442a-a6f2-c83502b51f0f-kube-api-access-27xf9\") pod \"cert-manager-cainjector-559cdd8c87-6nlsj\" (UID: \"f81bc1c6-4133-442a-a6f2-c83502b51f0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.682064 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.682005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xf9\" (UniqueName: \"kubernetes.io/projected/f81bc1c6-4133-442a-a6f2-c83502b51f0f-kube-api-access-27xf9\") pod \"cert-manager-cainjector-559cdd8c87-6nlsj\" (UID: \"f81bc1c6-4133-442a-a6f2-c83502b51f0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.682233 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.682108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f81bc1c6-4133-442a-a6f2-c83502b51f0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6nlsj\" (UID: \"f81bc1c6-4133-442a-a6f2-c83502b51f0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.690292 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.690235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f81bc1c6-4133-442a-a6f2-c83502b51f0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6nlsj\" (UID: \"f81bc1c6-4133-442a-a6f2-c83502b51f0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.690392 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.690375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xf9\" (UniqueName: \"kubernetes.io/projected/f81bc1c6-4133-442a-a6f2-c83502b51f0f-kube-api-access-27xf9\") pod \"cert-manager-cainjector-559cdd8c87-6nlsj\" (UID: \"f81bc1c6-4133-442a-a6f2-c83502b51f0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.749899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.749863    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"
Jul 24 01:34:37.876744 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:37.876724    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj"]
Jul 24 01:34:37.878793 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:37.878767    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf81bc1c6_4133_442a_a6f2_c83502b51f0f.slice/crio-9cb81b7ad7836fe1641f2c394b3a83894ad38444ea16d075402470555531870d WatchSource:0}: Error finding container 9cb81b7ad7836fe1641f2c394b3a83894ad38444ea16d075402470555531870d: Status 404 returned error can't find the container with id 9cb81b7ad7836fe1641f2c394b3a83894ad38444ea16d075402470555531870d
Jul 24 01:34:38.104649 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:38.104613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj" event={"ID":"f81bc1c6-4133-442a-a6f2-c83502b51f0f","Type":"ContainerStarted","Data":"9cb81b7ad7836fe1641f2c394b3a83894ad38444ea16d075402470555531870d"}
Jul 24 01:34:38.106558 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:38.106527    2564 generic.go:356] "Generic (PLEG): container finished" podID="308cb2bd-bf7b-471b-a122-14899b8621da" containerID="409d0b73a3351c692f8bfaabcf57ebaa92581c3d649e6992ddfa7c870911e643" exitCode=0
Jul 24 01:34:38.106672 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:38.106583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4" event={"ID":"308cb2bd-bf7b-471b-a122-14899b8621da","Type":"ContainerDied","Data":"409d0b73a3351c692f8bfaabcf57ebaa92581c3d649e6992ddfa7c870911e643"}
Jul 24 01:34:38.107863 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:38.107836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn" event={"ID":"35f10a4f-8e54-4c5f-b793-ce03f9247ca7","Type":"ContainerStarted","Data":"68f5f23f66a97d37b39cd6de6d4d72b799ae209f0e793df35756f4f65ce958b6"}
Jul 24 01:34:39.116886 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:39.116848    2564 generic.go:356] "Generic (PLEG): container finished" podID="308cb2bd-bf7b-471b-a122-14899b8621da" containerID="782ca71cacf24c5cd73844dd843b7433aa7cf5bdb195f5fdd554f1ac5243ec80" exitCode=0
Jul 24 01:34:39.117424 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:39.116925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4" event={"ID":"308cb2bd-bf7b-471b-a122-14899b8621da","Type":"ContainerDied","Data":"782ca71cacf24c5cd73844dd843b7433aa7cf5bdb195f5fdd554f1ac5243ec80"}
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.285886    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.286051    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.290286    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.292652    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.292998    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.296349    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.406361    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-util\") pod \"308cb2bd-bf7b-471b-a122-14899b8621da\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") "
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.406527    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxjvn\" (UniqueName: \"kubernetes.io/projected/308cb2bd-bf7b-471b-a122-14899b8621da-kube-api-access-dxjvn\") pod \"308cb2bd-bf7b-471b-a122-14899b8621da\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") "
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.406591    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-bundle\") pod \"308cb2bd-bf7b-471b-a122-14899b8621da\" (UID: \"308cb2bd-bf7b-471b-a122-14899b8621da\") "
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.406897    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-bundle" (OuterVolumeSpecName: "bundle") pod "308cb2bd-bf7b-471b-a122-14899b8621da" (UID: "308cb2bd-bf7b-471b-a122-14899b8621da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.408696    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308cb2bd-bf7b-471b-a122-14899b8621da-kube-api-access-dxjvn" (OuterVolumeSpecName: "kube-api-access-dxjvn") pod "308cb2bd-bf7b-471b-a122-14899b8621da" (UID: "308cb2bd-bf7b-471b-a122-14899b8621da"). InnerVolumeSpecName "kube-api-access-dxjvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.411704    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-util" (OuterVolumeSpecName: "util") pod "308cb2bd-bf7b-471b-a122-14899b8621da" (UID: "308cb2bd-bf7b-471b-a122-14899b8621da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.508206    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjvn\" (UniqueName: \"kubernetes.io/projected/308cb2bd-bf7b-471b-a122-14899b8621da-kube-api-access-dxjvn\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.508233    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:40.588454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:40.508244    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/308cb2bd-bf7b-471b-a122-14899b8621da-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:41.128762 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.128720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj" event={"ID":"f81bc1c6-4133-442a-a6f2-c83502b51f0f","Type":"ContainerStarted","Data":"2b54e0c0ddab078b235ce235bcdc78cb16fbacbec3c65f133cfe65b4e85e38e6"}
Jul 24 01:34:41.130593 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.130567    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4"
Jul 24 01:34:41.130734 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.130564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7d5g4" event={"ID":"308cb2bd-bf7b-471b-a122-14899b8621da","Type":"ContainerDied","Data":"04affb399e92768e52284fd4994f0a182096b47dc75ab676cf5ba3f12115ac71"}
Jul 24 01:34:41.130734 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.130685    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04affb399e92768e52284fd4994f0a182096b47dc75ab676cf5ba3f12115ac71"
Jul 24 01:34:41.132148 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.132122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn" event={"ID":"35f10a4f-8e54-4c5f-b793-ce03f9247ca7","Type":"ContainerStarted","Data":"8dc730c873a962f8d6e2daf37f04e6fabd8b55b086c4dc4acbf5ef0663153827"}
Jul 24 01:34:41.132282 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.132245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:41.143653 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.143610    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nlsj" podStartSLOduration=1.482474034 podStartE2EDuration="4.143596545s" totalImagesPullingTime="2.661122511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:34:41.14213697 +0000 UTC m=+301.564203804" watchObservedRunningTime="2026-07-24 01:34:41.143596545 +0000 UTC m=+301.565663381"
Jul 24 01:34:41.162540 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:41.162491    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn" podStartSLOduration=1.978359233 podStartE2EDuration="5.162474017s" totalImagesPullingTime="3.184114784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:34:41.160881385 +0000 UTC m=+301.582948221" watchObservedRunningTime="2026-07-24 01:34:41.162474017 +0000 UTC m=+301.584540853"
Jul 24 01:34:45.810816 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.810781    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lmc97"]
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811145    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="extract"
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811158    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="extract"
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811174    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="pull"
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811179    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="pull"
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811187    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="util"
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811193    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="util"
Jul 24 01:34:45.811312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.811250    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="308cb2bd-bf7b-471b-a122-14899b8621da" containerName="extract"
Jul 24 01:34:45.813279 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.813260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:45.815390 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.815367    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wwvn4\""
Jul 24 01:34:45.823388 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.823363    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lmc97"]
Jul 24 01:34:45.954986 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.954951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2gn\" (UniqueName: \"kubernetes.io/projected/5094cb57-57f9-4f3b-b6a3-c44cf878ed25-kube-api-access-bw2gn\") pod \"cert-manager-55b4d954d5-lmc97\" (UID: \"5094cb57-57f9-4f3b-b6a3-c44cf878ed25\") " pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:45.955207 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:45.955048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5094cb57-57f9-4f3b-b6a3-c44cf878ed25-bound-sa-token\") pod \"cert-manager-55b4d954d5-lmc97\" (UID: \"5094cb57-57f9-4f3b-b6a3-c44cf878ed25\") " pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:46.056017 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:46.055981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2gn\" (UniqueName: \"kubernetes.io/projected/5094cb57-57f9-4f3b-b6a3-c44cf878ed25-kube-api-access-bw2gn\") pod \"cert-manager-55b4d954d5-lmc97\" (UID: \"5094cb57-57f9-4f3b-b6a3-c44cf878ed25\") " pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:46.056225 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:46.056073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5094cb57-57f9-4f3b-b6a3-c44cf878ed25-bound-sa-token\") pod \"cert-manager-55b4d954d5-lmc97\" (UID: \"5094cb57-57f9-4f3b-b6a3-c44cf878ed25\") " pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:46.064670 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:46.064605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5094cb57-57f9-4f3b-b6a3-c44cf878ed25-bound-sa-token\") pod \"cert-manager-55b4d954d5-lmc97\" (UID: \"5094cb57-57f9-4f3b-b6a3-c44cf878ed25\") " pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:46.064900 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:46.064878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2gn\" (UniqueName: \"kubernetes.io/projected/5094cb57-57f9-4f3b-b6a3-c44cf878ed25-kube-api-access-bw2gn\") pod \"cert-manager-55b4d954d5-lmc97\" (UID: \"5094cb57-57f9-4f3b-b6a3-c44cf878ed25\") " pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:46.122681 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:46.122644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lmc97"
Jul 24 01:34:46.252171 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:46.252142    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lmc97"]
Jul 24 01:34:46.254627 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:46.254596    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5094cb57_57f9_4f3b_b6a3_c44cf878ed25.slice/crio-65467f530c3630ddeb394ad3907f1a8f8bb8fb6c6e4bc581d7860c20ae65ca26 WatchSource:0}: Error finding container 65467f530c3630ddeb394ad3907f1a8f8bb8fb6c6e4bc581d7860c20ae65ca26: Status 404 returned error can't find the container with id 65467f530c3630ddeb394ad3907f1a8f8bb8fb6c6e4bc581d7860c20ae65ca26
Jul 24 01:34:47.138514 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:47.138482    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f7hrn"
Jul 24 01:34:47.161248 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:47.161214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lmc97" event={"ID":"5094cb57-57f9-4f3b-b6a3-c44cf878ed25","Type":"ContainerStarted","Data":"4fc05957b414d3eb29d8aaf9c086f58a45952a55123bd674d4af4df2db3aea7e"}
Jul 24 01:34:47.161415 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:47.161257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lmc97" event={"ID":"5094cb57-57f9-4f3b-b6a3-c44cf878ed25","Type":"ContainerStarted","Data":"65467f530c3630ddeb394ad3907f1a8f8bb8fb6c6e4bc581d7860c20ae65ca26"}
Jul 24 01:34:47.177855 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:47.177794    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lmc97" podStartSLOduration=2.17777376 podStartE2EDuration="2.17777376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:34:47.177316529 +0000 UTC m=+307.599383367" watchObservedRunningTime="2026-07-24 01:34:47.17777376 +0000 UTC m=+307.599840597"
Jul 24 01:34:51.091051 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.090993    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"]
Jul 24 01:34:51.095804 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.095786    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.098056 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.098009    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 01:34:51.098156 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.098101    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 01:34:51.098792 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.098778    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5sgzw\""
Jul 24 01:34:51.101620 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.101593    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"]
Jul 24 01:34:51.199617 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.199581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.199800 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.199654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gqw\" (UniqueName: \"kubernetes.io/projected/daa1f397-1b88-4a4d-b91d-35d2663c1c57-kube-api-access-t4gqw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.199800 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.199710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.300789 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.300741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.300963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.300831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.300963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.300895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gqw\" (UniqueName: \"kubernetes.io/projected/daa1f397-1b88-4a4d-b91d-35d2663c1c57-kube-api-access-t4gqw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.301261 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.301241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.301304 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.301250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.308843 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.308814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gqw\" (UniqueName: \"kubernetes.io/projected/daa1f397-1b88-4a4d-b91d-35d2663c1c57-kube-api-access-t4gqw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.407059 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.406955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:51.540170 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:51.540142    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"]
Jul 24 01:34:51.542648 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:34:51.542618    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa1f397_1b88_4a4d_b91d_35d2663c1c57.slice/crio-3e8ff72ba1971e11b025040368849b0aa103457dd556cbbcc0442376922f1916 WatchSource:0}: Error finding container 3e8ff72ba1971e11b025040368849b0aa103457dd556cbbcc0442376922f1916: Status 404 returned error can't find the container with id 3e8ff72ba1971e11b025040368849b0aa103457dd556cbbcc0442376922f1916
Jul 24 01:34:52.184683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:52.184650    2564 generic.go:356] "Generic (PLEG): container finished" podID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerID="9e7cba8e325b1024176a50613439cf58f73c66fd8514b864140ede3b5c2fffe2" exitCode=0
Jul 24 01:34:52.185082 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:52.184730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh" event={"ID":"daa1f397-1b88-4a4d-b91d-35d2663c1c57","Type":"ContainerDied","Data":"9e7cba8e325b1024176a50613439cf58f73c66fd8514b864140ede3b5c2fffe2"}
Jul 24 01:34:52.185082 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:52.184767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh" event={"ID":"daa1f397-1b88-4a4d-b91d-35d2663c1c57","Type":"ContainerStarted","Data":"3e8ff72ba1971e11b025040368849b0aa103457dd556cbbcc0442376922f1916"}
Jul 24 01:34:52.185252 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:52.185236    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 01:34:54.196744 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:54.196704    2564 generic.go:356] "Generic (PLEG): container finished" podID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerID="d855208dc5907a10fefb0d9bff846927a970342fd4b88a299fdb837fc4890730" exitCode=0
Jul 24 01:34:54.197364 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:54.196759    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh" event={"ID":"daa1f397-1b88-4a4d-b91d-35d2663c1c57","Type":"ContainerDied","Data":"d855208dc5907a10fefb0d9bff846927a970342fd4b88a299fdb837fc4890730"}
Jul 24 01:34:55.203414 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:55.203373    2564 generic.go:356] "Generic (PLEG): container finished" podID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerID="e5b7765e67312e029b1a65a1bb72bae312ca7a840493561e0ff14e91e41e2be0" exitCode=0
Jul 24 01:34:55.203820 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:55.203456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh" event={"ID":"daa1f397-1b88-4a4d-b91d-35d2663c1c57","Type":"ContainerDied","Data":"e5b7765e67312e029b1a65a1bb72bae312ca7a840493561e0ff14e91e41e2be0"}
Jul 24 01:34:56.349427 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.349402    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:56.447904 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.447871    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-util\") pod \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") "
Jul 24 01:34:56.448096 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.447918    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4gqw\" (UniqueName: \"kubernetes.io/projected/daa1f397-1b88-4a4d-b91d-35d2663c1c57-kube-api-access-t4gqw\") pod \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") "
Jul 24 01:34:56.448096 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.447956    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-bundle\") pod \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\" (UID: \"daa1f397-1b88-4a4d-b91d-35d2663c1c57\") "
Jul 24 01:34:56.448771 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.448741    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-bundle" (OuterVolumeSpecName: "bundle") pod "daa1f397-1b88-4a4d-b91d-35d2663c1c57" (UID: "daa1f397-1b88-4a4d-b91d-35d2663c1c57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:34:56.450197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.450171    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa1f397-1b88-4a4d-b91d-35d2663c1c57-kube-api-access-t4gqw" (OuterVolumeSpecName: "kube-api-access-t4gqw") pod "daa1f397-1b88-4a4d-b91d-35d2663c1c57" (UID: "daa1f397-1b88-4a4d-b91d-35d2663c1c57"). InnerVolumeSpecName "kube-api-access-t4gqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:34:56.453601 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.453568    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-util" (OuterVolumeSpecName: "util") pod "daa1f397-1b88-4a4d-b91d-35d2663c1c57" (UID: "daa1f397-1b88-4a4d-b91d-35d2663c1c57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:34:56.548899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.548816    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:56.548899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.548846    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4gqw\" (UniqueName: \"kubernetes.io/projected/daa1f397-1b88-4a4d-b91d-35d2663c1c57-kube-api-access-t4gqw\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:56.548899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:56.548858    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daa1f397-1b88-4a4d-b91d-35d2663c1c57-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:34:57.215118 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:57.215092    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh"
Jul 24 01:34:57.215286 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:57.215090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkhtqh" event={"ID":"daa1f397-1b88-4a4d-b91d-35d2663c1c57","Type":"ContainerDied","Data":"3e8ff72ba1971e11b025040368849b0aa103457dd556cbbcc0442376922f1916"}
Jul 24 01:34:57.215286 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:34:57.215214    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e8ff72ba1971e11b025040368849b0aa103457dd556cbbcc0442376922f1916"
Jul 24 01:35:09.015809 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.015778    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"]
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016128    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="pull"
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016143    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="pull"
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016155    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="extract"
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016161    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="extract"
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016169    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="util"
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016175    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="util"
Jul 24 01:35:09.016385 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.016246    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa1f397-1b88-4a4d-b91d-35d2663c1c57" containerName="extract"
Jul 24 01:35:09.020546 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.020527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.022850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.022814    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 01:35:09.022850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.022846    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 01:35:09.023104 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.023084    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 01:35:09.023350 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.023334    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 01:35:09.023403 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.023366    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ztd2k\""
Jul 24 01:35:09.035803 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.035779    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"]
Jul 24 01:35:09.040118 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.040085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b89af98-48a7-4539-8b9c-222585ec8154-tmp\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.040218 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.040133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4xv\" (UniqueName: \"kubernetes.io/projected/3b89af98-48a7-4539-8b9c-222585ec8154-kube-api-access-hs4xv\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.040268 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.040227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b89af98-48a7-4539-8b9c-222585ec8154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.040309 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.040280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b89af98-48a7-4539-8b9c-222585ec8154-webhook-cert\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.040355 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.040322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b89af98-48a7-4539-8b9c-222585ec8154-manifests\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.140779 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.140738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b89af98-48a7-4539-8b9c-222585ec8154-tmp\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.140981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.140788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xv\" (UniqueName: \"kubernetes.io/projected/3b89af98-48a7-4539-8b9c-222585ec8154-kube-api-access-hs4xv\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.140981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.140841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b89af98-48a7-4539-8b9c-222585ec8154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.140981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.140868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b89af98-48a7-4539-8b9c-222585ec8154-webhook-cert\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.140981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.140898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b89af98-48a7-4539-8b9c-222585ec8154-manifests\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.141343 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.141314    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b89af98-48a7-4539-8b9c-222585ec8154-manifests\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.143144 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.143120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b89af98-48a7-4539-8b9c-222585ec8154-tmp\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.143441 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.143419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b89af98-48a7-4539-8b9c-222585ec8154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.143497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.143467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b89af98-48a7-4539-8b9c-222585ec8154-webhook-cert\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.148788 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.148755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4xv\" (UniqueName: \"kubernetes.io/projected/3b89af98-48a7-4539-8b9c-222585ec8154-kube-api-access-hs4xv\") pod \"opendatahub-operator-controller-manager-59c5678768-lxb4h\" (UID: \"3b89af98-48a7-4539-8b9c-222585ec8154\") " pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.330578 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.330495    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:09.470354 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:09.470313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"]
Jul 24 01:35:09.473934 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:35:09.473902    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b89af98_48a7_4539_8b9c_222585ec8154.slice/crio-7035f8555314d4a978b026dd26e171b717169e3236f7b5e9bf1079e31172f662 WatchSource:0}: Error finding container 7035f8555314d4a978b026dd26e171b717169e3236f7b5e9bf1079e31172f662: Status 404 returned error can't find the container with id 7035f8555314d4a978b026dd26e171b717169e3236f7b5e9bf1079e31172f662
Jul 24 01:35:10.277020 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:10.276974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h" event={"ID":"3b89af98-48a7-4539-8b9c-222585ec8154","Type":"ContainerStarted","Data":"7035f8555314d4a978b026dd26e171b717169e3236f7b5e9bf1079e31172f662"}
Jul 24 01:35:13.294211 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:13.294119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h" event={"ID":"3b89af98-48a7-4539-8b9c-222585ec8154","Type":"ContainerStarted","Data":"de029cb6f7acfc90e0bf5b6c39118a114c8c7f883de3ad31375324491d5ccd81"}
Jul 24 01:35:14.300851 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:14.300816    2564 generic.go:356] "Generic (PLEG): container finished" podID="3b89af98-48a7-4539-8b9c-222585ec8154" containerID="de029cb6f7acfc90e0bf5b6c39118a114c8c7f883de3ad31375324491d5ccd81" exitCode=0
Jul 24 01:35:14.301249 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:14.300906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h" event={"ID":"3b89af98-48a7-4539-8b9c-222585ec8154","Type":"ContainerDied","Data":"de029cb6f7acfc90e0bf5b6c39118a114c8c7f883de3ad31375324491d5ccd81"}
Jul 24 01:35:15.307461 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:15.307417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h" event={"ID":"3b89af98-48a7-4539-8b9c-222585ec8154","Type":"ContainerStarted","Data":"ba0e264ab6094526f7a7ab99ce4a81393f15831806e57d6c4845c6c6b9fe08b9"}
Jul 24 01:35:15.307922 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:15.307536    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:15.328747 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:15.328696    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h" podStartSLOduration=2.307811631 podStartE2EDuration="7.328677927s" totalImagesPullingTime="3.974206754s" totalInitContainerRuntime="1.046659542s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:35:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:35:15.326408812 +0000 UTC m=+335.748475684" watchObservedRunningTime="2026-07-24 01:35:15.328677927 +0000 UTC m=+335.750744757"
Jul 24 01:35:26.315374 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:26.315342    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59c5678768-lxb4h"
Jul 24 01:35:28.756283 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.756238    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"]
Jul 24 01:35:28.761889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.761868    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.765412 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.765383    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 01:35:28.767426 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.767399    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 01:35:28.768974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.768499    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 01:35:28.768974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.768514    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 01:35:28.768974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.768518    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9pcv6\""
Jul 24 01:35:28.776121 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.776099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"]
Jul 24 01:35:28.806327 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.806295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58b537b9-80f4-48c8-8d79-32605a755b58-tmp\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.806503 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.806345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b537b9-80f4-48c8-8d79-32605a755b58-tls-certs\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.806503 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.806413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zt5\" (UniqueName: \"kubernetes.io/projected/58b537b9-80f4-48c8-8d79-32605a755b58-kube-api-access-29zt5\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.890447 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.890407    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sn9r6"]
Jul 24 01:35:28.893147 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.893123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:28.895436 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.895405    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-g6c4p\""
Jul 24 01:35:28.895544 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.895487    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 01:35:28.906448 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.906420    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sn9r6"]
Jul 24 01:35:28.907713 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.907663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58b537b9-80f4-48c8-8d79-32605a755b58-tmp\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.907836 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.907773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b537b9-80f4-48c8-8d79-32605a755b58-tls-certs\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.907914 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.907835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zt5\" (UniqueName: \"kubernetes.io/projected/58b537b9-80f4-48c8-8d79-32605a755b58-kube-api-access-29zt5\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.911584 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.911563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58b537b9-80f4-48c8-8d79-32605a755b58-tmp\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.911751 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.911625    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b537b9-80f4-48c8-8d79-32605a755b58-tls-certs\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:28.921465 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:28.921439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zt5\" (UniqueName: \"kubernetes.io/projected/58b537b9-80f4-48c8-8d79-32605a755b58-kube-api-access-29zt5\") pod \"kube-auth-proxy-8544c7785d-7xqm5\" (UID: \"58b537b9-80f4-48c8-8d79-32605a755b58\") " pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:29.009235 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.009151    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779f50f6-88e4-4731-ad1d-70901f32e0a9-cert\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.009235 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.009197    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzqf\" (UniqueName: \"kubernetes.io/projected/779f50f6-88e4-4731-ad1d-70901f32e0a9-kube-api-access-hzzqf\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.072141 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.072104    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"
Jul 24 01:35:29.110495 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.110454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzqf\" (UniqueName: \"kubernetes.io/projected/779f50f6-88e4-4731-ad1d-70901f32e0a9-kube-api-access-hzzqf\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.110668 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.110619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779f50f6-88e4-4731-ad1d-70901f32e0a9-cert\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.110771 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:35:29.110755    2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 01:35:29.110842 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:35:29.110830    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779f50f6-88e4-4731-ad1d-70901f32e0a9-cert podName:779f50f6-88e4-4731-ad1d-70901f32e0a9 nodeName:}" failed. No retries permitted until 2026-07-24 01:35:29.61080821 +0000 UTC m=+350.032875026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/779f50f6-88e4-4731-ad1d-70901f32e0a9-cert") pod "odh-model-controller-7998c67dc4-sn9r6" (UID: "779f50f6-88e4-4731-ad1d-70901f32e0a9") : secret "odh-model-controller-webhook-cert" not found
Jul 24 01:35:29.130315 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.130286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzqf\" (UniqueName: \"kubernetes.io/projected/779f50f6-88e4-4731-ad1d-70901f32e0a9-kube-api-access-hzzqf\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.230403 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.230371    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5"]
Jul 24 01:35:29.233759 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:35:29.233733    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b537b9_80f4_48c8_8d79_32605a755b58.slice/crio-8869855d7a430b8156c3438d1646429e2c90f464a1c0d4299af54ea6d26a4ae8 WatchSource:0}: Error finding container 8869855d7a430b8156c3438d1646429e2c90f464a1c0d4299af54ea6d26a4ae8: Status 404 returned error can't find the container with id 8869855d7a430b8156c3438d1646429e2c90f464a1c0d4299af54ea6d26a4ae8
Jul 24 01:35:29.376649 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.376567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5" event={"ID":"58b537b9-80f4-48c8-8d79-32605a755b58","Type":"ContainerStarted","Data":"8869855d7a430b8156c3438d1646429e2c90f464a1c0d4299af54ea6d26a4ae8"}
Jul 24 01:35:29.615441 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.615402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779f50f6-88e4-4731-ad1d-70901f32e0a9-cert\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.617834 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.617813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779f50f6-88e4-4731-ad1d-70901f32e0a9-cert\") pod \"odh-model-controller-7998c67dc4-sn9r6\" (UID: \"779f50f6-88e4-4731-ad1d-70901f32e0a9\") " pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.779077 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.779044    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddw4l"]
Jul 24 01:35:29.782084 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.782062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:29.785245 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.785223    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 24 01:35:29.785375 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.785350    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9z5qx\""
Jul 24 01:35:29.795700 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.795666    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddw4l"]
Jul 24 01:35:29.805366 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.805333    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:29.919842 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.919791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wpq\" (UniqueName: \"kubernetes.io/projected/0d964262-f632-47c5-8c90-aa3be7424715-kube-api-access-r7wpq\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:29.920203 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.919951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d964262-f632-47c5-8c90-aa3be7424715-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:29.942122 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:29.942098    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sn9r6"]
Jul 24 01:35:29.944402 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:35:29.944375    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779f50f6_88e4_4731_ad1d_70901f32e0a9.slice/crio-2d0074a75a66749b03cd3f1a4852a897bfbc80ab67ea16414f9b87db85c62460 WatchSource:0}: Error finding container 2d0074a75a66749b03cd3f1a4852a897bfbc80ab67ea16414f9b87db85c62460: Status 404 returned error can't find the container with id 2d0074a75a66749b03cd3f1a4852a897bfbc80ab67ea16414f9b87db85c62460
Jul 24 01:35:30.021134 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.021098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wpq\" (UniqueName: \"kubernetes.io/projected/0d964262-f632-47c5-8c90-aa3be7424715-kube-api-access-r7wpq\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:30.021309 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.021158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d964262-f632-47c5-8c90-aa3be7424715-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:30.021309 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:35:30.021255    2564 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 24 01:35:30.021392 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:35:30.021319    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d964262-f632-47c5-8c90-aa3be7424715-tls-certs podName:0d964262-f632-47c5-8c90-aa3be7424715 nodeName:}" failed. No retries permitted until 2026-07-24 01:35:30.521301665 +0000 UTC m=+350.943368479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0d964262-f632-47c5-8c90-aa3be7424715-tls-certs") pod "model-serving-api-54d89bd5fc-ddw4l" (UID: "0d964262-f632-47c5-8c90-aa3be7424715") : secret "model-serving-api-tls" not found
Jul 24 01:35:30.035763 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.035693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wpq\" (UniqueName: \"kubernetes.io/projected/0d964262-f632-47c5-8c90-aa3be7424715-kube-api-access-r7wpq\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:30.384063 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.383951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6" event={"ID":"779f50f6-88e4-4731-ad1d-70901f32e0a9","Type":"ContainerStarted","Data":"2d0074a75a66749b03cd3f1a4852a897bfbc80ab67ea16414f9b87db85c62460"}
Jul 24 01:35:30.525577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.525540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d964262-f632-47c5-8c90-aa3be7424715-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:30.528151 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.528127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d964262-f632-47c5-8c90-aa3be7424715-tls-certs\") pod \"model-serving-api-54d89bd5fc-ddw4l\" (UID: \"0d964262-f632-47c5-8c90-aa3be7424715\") " pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:30.696381 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.696345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:30.878788 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:30.878711    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ddw4l"]
Jul 24 01:35:31.393414 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:31.393368    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l" event={"ID":"0d964262-f632-47c5-8c90-aa3be7424715","Type":"ContainerStarted","Data":"949eb19d62693884a6f55721d24b46fd1dbd771bdf46f05c3bc625354ca5425c"}
Jul 24 01:35:34.411577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.411539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l" event={"ID":"0d964262-f632-47c5-8c90-aa3be7424715","Type":"ContainerStarted","Data":"5431112a26b02b76ed5061e3156c64cfdc69a5da36a5fbbed1be70108827c96f"}
Jul 24 01:35:34.412127 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.411630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:34.413425 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.413391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6" event={"ID":"779f50f6-88e4-4731-ad1d-70901f32e0a9","Type":"ContainerStarted","Data":"99f19585f9ece76648ed5df0d42e701d2cd404bb436c52f17c6519ae6bf54eda"}
Jul 24 01:35:34.413546 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.413452    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:34.415383 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.415361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5" event={"ID":"58b537b9-80f4-48c8-8d79-32605a755b58","Type":"ContainerStarted","Data":"51de49a594e63cb816c1dbad7d71c32bf55352ea76b2806ffb0aeb68b83c5ca5"}
Jul 24 01:35:34.427100 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.427001    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l" podStartSLOduration=2.441267876 podStartE2EDuration="5.426985714s" totalImagesPullingTime="2.985717838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:35:34.426075147 +0000 UTC m=+354.848142002" watchObservedRunningTime="2026-07-24 01:35:34.426985714 +0000 UTC m=+354.849052551"
Jul 24 01:35:34.445264 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.445206    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8544c7785d-7xqm5" podStartSLOduration=2.230760708 podStartE2EDuration="6.445192925s" totalImagesPullingTime="4.214432217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:35:34.443998262 +0000 UTC m=+354.866065111" watchObservedRunningTime="2026-07-24 01:35:34.445192925 +0000 UTC m=+354.867259761"
Jul 24 01:35:34.460772 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:34.460723    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6" podStartSLOduration=2.584501877 podStartE2EDuration="6.460703432s" totalImagesPullingTime="3.876201555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:35:34.460082231 +0000 UTC m=+354.882149092" watchObservedRunningTime="2026-07-24 01:35:34.460703432 +0000 UTC m=+354.882770268"
Jul 24 01:35:36.132789 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.132754    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"]
Jul 24 01:35:36.136212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.136194    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.138754 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.138733    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 01:35:36.138754 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.138746    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 01:35:36.138936 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.138750    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 01:35:36.138936 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.138737    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tlwvw\""
Jul 24 01:35:36.139055 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.139018    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 01:35:36.157888 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.157853    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"]
Jul 24 01:35:36.280951 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.280909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.281123 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.280983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.281123 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.281085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.281231 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.281141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.281231 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.281178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c323c-8c8c-48df-9563-0ab7ea6b3868-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.281231 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.281225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.281355 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.281250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/b22c323c-8c8c-48df-9563-0ab7ea6b3868-kube-api-access-csl6r\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382323 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c323c-8c8c-48df-9563-0ab7ea6b3868-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/b22c323c-8c8c-48df-9563-0ab7ea6b3868-kube-api-access-csl6r\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382705 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.382705 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.382552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.383467 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.383396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.384876 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.384838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.384992 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.384911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.384992 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.384934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c323c-8c8c-48df-9563-0ab7ea6b3868-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.384992 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.384952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.395558 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.395538    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c323c-8c8c-48df-9563-0ab7ea6b3868-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.398749 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.398732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/b22c323c-8c8c-48df-9563-0ab7ea6b3868-kube-api-access-csl6r\") pod \"istiod-openshift-gateway-5677ccb49b-msljx\" (UID: \"b22c323c-8c8c-48df-9563-0ab7ea6b3868\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.447242 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.447206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:36.584988 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:36.584954    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"]
Jul 24 01:35:36.586465 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:35:36.586433    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22c323c_8c8c_48df_9563_0ab7ea6b3868.slice/crio-f25f3762556e0154e70b7c9226ad67650e1376e93992ae4016643079dc664022 WatchSource:0}: Error finding container f25f3762556e0154e70b7c9226ad67650e1376e93992ae4016643079dc664022: Status 404 returned error can't find the container with id f25f3762556e0154e70b7c9226ad67650e1376e93992ae4016643079dc664022
Jul 24 01:35:37.431894 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:37.431845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx" event={"ID":"b22c323c-8c8c-48df-9563-0ab7ea6b3868","Type":"ContainerStarted","Data":"f25f3762556e0154e70b7c9226ad67650e1376e93992ae4016643079dc664022"}
Jul 24 01:35:38.525273 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.525232    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"]
Jul 24 01:35:38.528728 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.528705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:38.533216 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.532991    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 24 01:35:38.533216 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.533116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v4pxl\""
Jul 24 01:35:38.542223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.542199    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"]
Jul 24 01:35:38.603068 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.603007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qfg\" (UniqueName: \"kubernetes.io/projected/5205735c-72f0-4f80-9938-665014506e2f-kube-api-access-s2qfg\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:38.603237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.603114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5205735c-72f0-4f80-9938-665014506e2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:38.703827 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.703788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qfg\" (UniqueName: \"kubernetes.io/projected/5205735c-72f0-4f80-9938-665014506e2f-kube-api-access-s2qfg\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:38.703989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.703857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5205735c-72f0-4f80-9938-665014506e2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:38.704069 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:35:38.704012    2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 24 01:35:38.704118 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:35:38.704107    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5205735c-72f0-4f80-9938-665014506e2f-cert podName:5205735c-72f0-4f80-9938-665014506e2f nodeName:}" failed. No retries permitted until 2026-07-24 01:35:39.204090024 +0000 UTC m=+359.626156838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5205735c-72f0-4f80-9938-665014506e2f-cert") pod "llmisvc-controller-manager-777c6bd7d5-tdqpq" (UID: "5205735c-72f0-4f80-9938-665014506e2f") : secret "llmisvc-webhook-server-cert" not found
Jul 24 01:35:38.717283 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:38.717247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qfg\" (UniqueName: \"kubernetes.io/projected/5205735c-72f0-4f80-9938-665014506e2f-kube-api-access-s2qfg\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:39.209899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:39.209856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5205735c-72f0-4f80-9938-665014506e2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:39.212782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:39.212741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5205735c-72f0-4f80-9938-665014506e2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tdqpq\" (UID: \"5205735c-72f0-4f80-9938-665014506e2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:39.440665 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:39.440633    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:39.601578 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:39.601536    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"]
Jul 24 01:35:40.438603 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:35:40.438551    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5205735c_72f0_4f80_9938_665014506e2f.slice/crio-630acca3c251a30cab76dcbfa1d30af7fd99db52fbad0f1cece4317580f72822 WatchSource:0}: Error finding container 630acca3c251a30cab76dcbfa1d30af7fd99db52fbad0f1cece4317580f72822: Status 404 returned error can't find the container with id 630acca3c251a30cab76dcbfa1d30af7fd99db52fbad0f1cece4317580f72822
Jul 24 01:35:40.453766 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:40.453730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq" event={"ID":"5205735c-72f0-4f80-9938-665014506e2f","Type":"ContainerStarted","Data":"630acca3c251a30cab76dcbfa1d30af7fd99db52fbad0f1cece4317580f72822"}
Jul 24 01:35:40.513994 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:40.513952    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:35:40.514122 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:40.514041    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:35:41.467167 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:41.466061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx" event={"ID":"b22c323c-8c8c-48df-9563-0ab7ea6b3868","Type":"ContainerStarted","Data":"014a9a44efb7ac742b56954b6813fa7995b848d956360d6e05d7a2987d3f8b84"}
Jul 24 01:35:41.467167 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:41.467107    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:41.468632 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:41.468584    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-msljx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 01:35:41.468780 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:41.468661    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx" podUID="b22c323c-8c8c-48df-9563-0ab7ea6b3868" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 01:35:41.490145 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:41.490084    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx" podStartSLOduration=1.564904463 podStartE2EDuration="5.490062622s" totalImagesPullingTime="3.925158159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:35:41.487146241 +0000 UTC m=+361.909213098" watchObservedRunningTime="2026-07-24 01:35:41.490062622 +0000 UTC m=+361.912129459"
Jul 24 01:35:42.471392 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:42.471349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-msljx"
Jul 24 01:35:44.483117 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:44.483083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq" event={"ID":"5205735c-72f0-4f80-9938-665014506e2f","Type":"ContainerStarted","Data":"b677460b94592e2b4045aa01e3b8f6395138c25ae117d31916f794d461fb42bb"}
Jul 24 01:35:44.483525 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:44.483390    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:35:45.423929 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:45.423892    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sn9r6"
Jul 24 01:35:45.431478 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:45.431448    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ddw4l"
Jul 24 01:35:45.441843 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:35:45.441793    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq" podStartSLOduration=4.164747677 podStartE2EDuration="7.441778389s" totalImagesPullingTime="3.277030712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:35:44.500046427 +0000 UTC m=+364.922113274" watchObservedRunningTime="2026-07-24 01:35:45.441778389 +0000 UTC m=+365.863845226"
Jul 24 01:36:15.490807 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:15.490774    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tdqpq"
Jul 24 01:36:45.665606 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.665570    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"]
Jul 24 01:36:45.668204 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.668182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.670476 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.670456    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 01:36:45.670476 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.670465    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 01:36:45.671147 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.671130    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5dwhk\""
Jul 24 01:36:45.676971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.676947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"]
Jul 24 01:36:45.800089 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.800019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsw8b\" (UniqueName: \"kubernetes.io/projected/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-kube-api-access-fsw8b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.800256 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.800161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.800256 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.800204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.901302 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.901268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsw8b\" (UniqueName: \"kubernetes.io/projected/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-kube-api-access-fsw8b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.901474 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.901348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.901474 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.901371    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.901745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.901728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.901828 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.901771    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.911776 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.911746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsw8b\" (UniqueName: \"kubernetes.io/projected/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-kube-api-access-fsw8b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:45.979710 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:45.979672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:46.115969 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.115944    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"]
Jul 24 01:36:46.118214 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:36:46.118174    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf33f2fe_f4d5_4d39_9d6c_78307c1d284f.slice/crio-1e607d04cb47098b721c7fc07fdf4da0791fbc2d2c794906bdbe9bd79c4f46e1 WatchSource:0}: Error finding container 1e607d04cb47098b721c7fc07fdf4da0791fbc2d2c794906bdbe9bd79c4f46e1: Status 404 returned error can't find the container with id 1e607d04cb47098b721c7fc07fdf4da0791fbc2d2c794906bdbe9bd79c4f46e1
Jul 24 01:36:46.263054 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.262969    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"]
Jul 24 01:36:46.265868 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.265849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.273929 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.273904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"]
Jul 24 01:36:46.407352 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.407317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkds\" (UniqueName: \"kubernetes.io/projected/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-kube-api-access-tmkds\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.407612 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.407379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.407612 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.407405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.508708 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.508674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkds\" (UniqueName: \"kubernetes.io/projected/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-kube-api-access-tmkds\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.508882 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.508735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.508882 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.508762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.509183 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.509166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.509225 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.509210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.516275 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.516213    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkds\" (UniqueName: \"kubernetes.io/projected/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-kube-api-access-tmkds\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.579636 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.579603    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:46.667810 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.667776    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"]
Jul 24 01:36:46.672767 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.672743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.678521 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.678490    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"]
Jul 24 01:36:46.728057 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.728012    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"]
Jul 24 01:36:46.729246 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:36:46.729217    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfc01fd_8856_4a6d_ac2a_141b0c2545f5.slice/crio-fecdc3af657a0593399a55a94fe7d795ffda474009afea345661b09120bf91a8 WatchSource:0}: Error finding container fecdc3af657a0593399a55a94fe7d795ffda474009afea345661b09120bf91a8: Status 404 returned error can't find the container with id fecdc3af657a0593399a55a94fe7d795ffda474009afea345661b09120bf91a8
Jul 24 01:36:46.794162 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.794135    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerID="e0c9c9faa25ce54270170d59f6a5f6d29fee455a5c89a9a3da9af97c57ef26c4" exitCode=0
Jul 24 01:36:46.794272 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.794222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw" event={"ID":"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f","Type":"ContainerDied","Data":"e0c9c9faa25ce54270170d59f6a5f6d29fee455a5c89a9a3da9af97c57ef26c4"}
Jul 24 01:36:46.794323 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.794290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw" event={"ID":"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f","Type":"ContainerStarted","Data":"1e607d04cb47098b721c7fc07fdf4da0791fbc2d2c794906bdbe9bd79c4f46e1"}
Jul 24 01:36:46.796120 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.796087    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882" event={"ID":"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5","Type":"ContainerStarted","Data":"fecdc3af657a0593399a55a94fe7d795ffda474009afea345661b09120bf91a8"}
Jul 24 01:36:46.812441 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.812408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.812578 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.812539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.812640 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.812589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdwx\" (UniqueName: \"kubernetes.io/projected/fd8be8f8-4127-4b32-8a63-12fa11d6065d-kube-api-access-zcdwx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.913287 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.913246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.913459 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.913338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.913459 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.913368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdwx\" (UniqueName: \"kubernetes.io/projected/fd8be8f8-4127-4b32-8a63-12fa11d6065d-kube-api-access-zcdwx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.913777 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.913749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.913823 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.913773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.921448 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.921422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdwx\" (UniqueName: \"kubernetes.io/projected/fd8be8f8-4127-4b32-8a63-12fa11d6065d-kube-api-access-zcdwx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:46.986746 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:46.986711    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:47.077481 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.077444    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"]
Jul 24 01:36:47.083019 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.082988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.087743 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.087678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"]
Jul 24 01:36:47.124099 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.124073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"]
Jul 24 01:36:47.126084 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:36:47.126055    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8be8f8_4127_4b32_8a63_12fa11d6065d.slice/crio-f169d753e66d51f43e7fd18badf3dac423339e08d25d228211ccc7a133417f8e WatchSource:0}: Error finding container f169d753e66d51f43e7fd18badf3dac423339e08d25d228211ccc7a133417f8e: Status 404 returned error can't find the container with id f169d753e66d51f43e7fd18badf3dac423339e08d25d228211ccc7a133417f8e
Jul 24 01:36:47.215781 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.215748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pct6\" (UniqueName: \"kubernetes.io/projected/cc00c075-0552-47df-8578-a5996bde6f76-kube-api-access-5pct6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.215971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.215816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.215971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.215929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.317367 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.317274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.317550 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.317374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pct6\" (UniqueName: \"kubernetes.io/projected/cc00c075-0552-47df-8578-a5996bde6f76-kube-api-access-5pct6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.317550 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.317532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.317775 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.317751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.317852 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.317834    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.325409 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.325384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pct6\" (UniqueName: \"kubernetes.io/projected/cc00c075-0552-47df-8578-a5996bde6f76-kube-api-access-5pct6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.396335 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.396299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:47.539804 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.539777    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"]
Jul 24 01:36:47.566221 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:36:47.566185    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc00c075_0552_47df_8578_a5996bde6f76.slice/crio-cd29dd6174adfd226e912b247a8dd89f07c977c81602b58a1fb4cae06bcf3dae WatchSource:0}: Error finding container cd29dd6174adfd226e912b247a8dd89f07c977c81602b58a1fb4cae06bcf3dae: Status 404 returned error can't find the container with id cd29dd6174adfd226e912b247a8dd89f07c977c81602b58a1fb4cae06bcf3dae
Jul 24 01:36:47.804423 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.804386    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerID="84d27d33ffe3afd2664721449501c385edf59accebcce101826214a8f8c69a2b" exitCode=0
Jul 24 01:36:47.804855 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.804472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw" event={"ID":"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f","Type":"ContainerDied","Data":"84d27d33ffe3afd2664721449501c385edf59accebcce101826214a8f8c69a2b"}
Jul 24 01:36:47.806131 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.806110    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc00c075-0552-47df-8578-a5996bde6f76" containerID="eb3b2f298d246c04d09ba78f034b4c6e4824dd660a0f83aa20e5670f2ce37cba" exitCode=0
Jul 24 01:36:47.806209 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.806189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw" event={"ID":"cc00c075-0552-47df-8578-a5996bde6f76","Type":"ContainerDied","Data":"eb3b2f298d246c04d09ba78f034b4c6e4824dd660a0f83aa20e5670f2ce37cba"}
Jul 24 01:36:47.806258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.806237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw" event={"ID":"cc00c075-0552-47df-8578-a5996bde6f76","Type":"ContainerStarted","Data":"cd29dd6174adfd226e912b247a8dd89f07c977c81602b58a1fb4cae06bcf3dae"}
Jul 24 01:36:47.811403 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.811379    2564 generic.go:356] "Generic (PLEG): container finished" podID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerID="9cd32df4d4656af99fc7d2e56628ba2fb17c05192977642baaecf5e067b4e675" exitCode=0
Jul 24 01:36:47.811515 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.811477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882" event={"ID":"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5","Type":"ContainerDied","Data":"9cd32df4d4656af99fc7d2e56628ba2fb17c05192977642baaecf5e067b4e675"}
Jul 24 01:36:47.813141 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.813119    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerID="462a266cb0729ea1d67754d49724f074c088c7532ae5189b23cc5483cf58c00b" exitCode=0
Jul 24 01:36:47.813258 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.813217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s" event={"ID":"fd8be8f8-4127-4b32-8a63-12fa11d6065d","Type":"ContainerDied","Data":"462a266cb0729ea1d67754d49724f074c088c7532ae5189b23cc5483cf58c00b"}
Jul 24 01:36:47.813375 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:47.813261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s" event={"ID":"fd8be8f8-4127-4b32-8a63-12fa11d6065d","Type":"ContainerStarted","Data":"f169d753e66d51f43e7fd18badf3dac423339e08d25d228211ccc7a133417f8e"}
Jul 24 01:36:48.820864 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:48.820829    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerID="74b4638553999b711d0887810e1fc6e06ad242c6b6fec8f6ab9fbe759872f7dd" exitCode=0
Jul 24 01:36:48.821259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:48.820916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw" event={"ID":"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f","Type":"ContainerDied","Data":"74b4638553999b711d0887810e1fc6e06ad242c6b6fec8f6ab9fbe759872f7dd"}
Jul 24 01:36:48.822870 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:48.822846    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc00c075-0552-47df-8578-a5996bde6f76" containerID="fc49bdc5afeb9166f983fabf0e9641c51e8f9c942e2ee5f49314b12d7a96b080" exitCode=0
Jul 24 01:36:48.822990 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:48.822931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw" event={"ID":"cc00c075-0552-47df-8578-a5996bde6f76","Type":"ContainerDied","Data":"fc49bdc5afeb9166f983fabf0e9641c51e8f9c942e2ee5f49314b12d7a96b080"}
Jul 24 01:36:49.830426 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:49.830391    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc00c075-0552-47df-8578-a5996bde6f76" containerID="39ecd5050fad015b29153b67a3b48a56b0534b996edc57b5d553130952e164b5" exitCode=0
Jul 24 01:36:49.830816 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:49.830477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw" event={"ID":"cc00c075-0552-47df-8578-a5996bde6f76","Type":"ContainerDied","Data":"39ecd5050fad015b29153b67a3b48a56b0534b996edc57b5d553130952e164b5"}
Jul 24 01:36:49.832381 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:49.832362    2564 generic.go:356] "Generic (PLEG): container finished" podID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerID="430a53b4dfbba88add616c14230ad54103b9a78feba9e6ca50d32053b2a79e76" exitCode=0
Jul 24 01:36:49.832469 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:49.832431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882" event={"ID":"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5","Type":"ContainerDied","Data":"430a53b4dfbba88add616c14230ad54103b9a78feba9e6ca50d32053b2a79e76"}
Jul 24 01:36:49.834531 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:49.834510    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerID="94ebabda38c337705609a957c9aff5c5778be7269fba7fda8a69ee7d6f708100" exitCode=0
Jul 24 01:36:49.834619 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:49.834587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s" event={"ID":"fd8be8f8-4127-4b32-8a63-12fa11d6065d","Type":"ContainerDied","Data":"94ebabda38c337705609a957c9aff5c5778be7269fba7fda8a69ee7d6f708100"}
Jul 24 01:36:50.089250 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.089181    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:50.145878 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.145846    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-util\") pod \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") "
Jul 24 01:36:50.146061 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.145894    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsw8b\" (UniqueName: \"kubernetes.io/projected/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-kube-api-access-fsw8b\") pod \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") "
Jul 24 01:36:50.146061 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.145927    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-bundle\") pod \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\" (UID: \"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f\") "
Jul 24 01:36:50.146486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.146453    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-bundle" (OuterVolumeSpecName: "bundle") pod "cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" (UID: "cf33f2fe-f4d5-4d39-9d6c-78307c1d284f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:50.148256 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.148229    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-kube-api-access-fsw8b" (OuterVolumeSpecName: "kube-api-access-fsw8b") pod "cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" (UID: "cf33f2fe-f4d5-4d39-9d6c-78307c1d284f"). InnerVolumeSpecName "kube-api-access-fsw8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:36:50.151495 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.151461    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-util" (OuterVolumeSpecName: "util") pod "cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" (UID: "cf33f2fe-f4d5-4d39-9d6c-78307c1d284f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:50.246624 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.246589    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:50.246624 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.246617    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsw8b\" (UniqueName: \"kubernetes.io/projected/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-kube-api-access-fsw8b\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:50.246624 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.246626    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf33f2fe-f4d5-4d39-9d6c-78307c1d284f-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:50.841701 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.841667    2564 generic.go:356] "Generic (PLEG): container finished" podID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerID="5d53efe61a315fe02081fe345aaf673f241efe8c7b90a844654dedf4fd17658c" exitCode=0
Jul 24 01:36:50.842160 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.841745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882" event={"ID":"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5","Type":"ContainerDied","Data":"5d53efe61a315fe02081fe345aaf673f241efe8c7b90a844654dedf4fd17658c"}
Jul 24 01:36:50.843769 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.843740    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerID="7336dd275482b6b765389136e7696e8708266dc1545efce0c977911c07317672" exitCode=0
Jul 24 01:36:50.843901 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.843831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s" event={"ID":"fd8be8f8-4127-4b32-8a63-12fa11d6065d","Type":"ContainerDied","Data":"7336dd275482b6b765389136e7696e8708266dc1545efce0c977911c07317672"}
Jul 24 01:36:50.845517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.845501    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw"
Jul 24 01:36:50.845602 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.845533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw" event={"ID":"cf33f2fe-f4d5-4d39-9d6c-78307c1d284f","Type":"ContainerDied","Data":"1e607d04cb47098b721c7fc07fdf4da0791fbc2d2c794906bdbe9bd79c4f46e1"}
Jul 24 01:36:50.845602 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.845562    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e607d04cb47098b721c7fc07fdf4da0791fbc2d2c794906bdbe9bd79c4f46e1"
Jul 24 01:36:50.979654 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:50.979630    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:51.053972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.053929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-util\") pod \"cc00c075-0552-47df-8578-a5996bde6f76\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") "
Jul 24 01:36:51.053972 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.053977    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pct6\" (UniqueName: \"kubernetes.io/projected/cc00c075-0552-47df-8578-a5996bde6f76-kube-api-access-5pct6\") pod \"cc00c075-0552-47df-8578-a5996bde6f76\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") "
Jul 24 01:36:51.054243 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.054004    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-bundle\") pod \"cc00c075-0552-47df-8578-a5996bde6f76\" (UID: \"cc00c075-0552-47df-8578-a5996bde6f76\") "
Jul 24 01:36:51.054421 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.054397    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-bundle" (OuterVolumeSpecName: "bundle") pod "cc00c075-0552-47df-8578-a5996bde6f76" (UID: "cc00c075-0552-47df-8578-a5996bde6f76"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:51.056140 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.056116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc00c075-0552-47df-8578-a5996bde6f76-kube-api-access-5pct6" (OuterVolumeSpecName: "kube-api-access-5pct6") pod "cc00c075-0552-47df-8578-a5996bde6f76" (UID: "cc00c075-0552-47df-8578-a5996bde6f76"). InnerVolumeSpecName "kube-api-access-5pct6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:36:51.059347 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.059322    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-util" (OuterVolumeSpecName: "util") pod "cc00c075-0552-47df-8578-a5996bde6f76" (UID: "cc00c075-0552-47df-8578-a5996bde6f76"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:51.155446 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.155353    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:51.155446 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.155389    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pct6\" (UniqueName: \"kubernetes.io/projected/cc00c075-0552-47df-8578-a5996bde6f76-kube-api-access-5pct6\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:51.155446 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.155402    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc00c075-0552-47df-8578-a5996bde6f76-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:51.852285 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.852251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw"
Jul 24 01:36:51.852285 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.852273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw" event={"ID":"cc00c075-0552-47df-8578-a5996bde6f76","Type":"ContainerDied","Data":"cd29dd6174adfd226e912b247a8dd89f07c977c81602b58a1fb4cae06bcf3dae"}
Jul 24 01:36:51.852700 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:51.852319    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd29dd6174adfd226e912b247a8dd89f07c977c81602b58a1fb4cae06bcf3dae"
Jul 24 01:36:52.020915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.020892    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:52.036420 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.036386    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:52.065095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.062567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcdwx\" (UniqueName: \"kubernetes.io/projected/fd8be8f8-4127-4b32-8a63-12fa11d6065d-kube-api-access-zcdwx\") pod \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") "
Jul 24 01:36:52.065095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.062629    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-util\") pod \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") "
Jul 24 01:36:52.065095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.062674    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-bundle\") pod \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\" (UID: \"fd8be8f8-4127-4b32-8a63-12fa11d6065d\") "
Jul 24 01:36:52.065095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.062722    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-bundle\") pod \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") "
Jul 24 01:36:52.065095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.062811    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-util\") pod \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") "
Jul 24 01:36:52.065095 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.062848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmkds\" (UniqueName: \"kubernetes.io/projected/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-kube-api-access-tmkds\") pod \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\" (UID: \"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5\") "
Jul 24 01:36:52.065704 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.065670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-bundle" (OuterVolumeSpecName: "bundle") pod "fd8be8f8-4127-4b32-8a63-12fa11d6065d" (UID: "fd8be8f8-4127-4b32-8a63-12fa11d6065d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:52.065839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.065808    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-bundle" (OuterVolumeSpecName: "bundle") pod "4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" (UID: "4dfc01fd-8856-4a6d-ac2a-141b0c2545f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:52.068139 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.068106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8be8f8-4127-4b32-8a63-12fa11d6065d-kube-api-access-zcdwx" (OuterVolumeSpecName: "kube-api-access-zcdwx") pod "fd8be8f8-4127-4b32-8a63-12fa11d6065d" (UID: "fd8be8f8-4127-4b32-8a63-12fa11d6065d"). InnerVolumeSpecName "kube-api-access-zcdwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:36:52.068139 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.068125    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-kube-api-access-tmkds" (OuterVolumeSpecName: "kube-api-access-tmkds") pod "4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" (UID: "4dfc01fd-8856-4a6d-ac2a-141b0c2545f5"). InnerVolumeSpecName "kube-api-access-tmkds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:36:52.073309 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.073261    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-util" (OuterVolumeSpecName: "util") pod "fd8be8f8-4127-4b32-8a63-12fa11d6065d" (UID: "fd8be8f8-4127-4b32-8a63-12fa11d6065d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:52.162588 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.162491    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-util" (OuterVolumeSpecName: "util") pod "4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" (UID: "4dfc01fd-8856-4a6d-ac2a-141b0c2545f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:36:52.164172 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.164146    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:52.164172 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.164169    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:52.164266 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.164180    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkds\" (UniqueName: \"kubernetes.io/projected/4dfc01fd-8856-4a6d-ac2a-141b0c2545f5-kube-api-access-tmkds\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:52.164266 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.164191    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdwx\" (UniqueName: \"kubernetes.io/projected/fd8be8f8-4127-4b32-8a63-12fa11d6065d-kube-api-access-zcdwx\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:52.164266 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.164199    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-util\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:52.164266 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.164207    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd8be8f8-4127-4b32-8a63-12fa11d6065d-bundle\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:36:52.859275 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.859243    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882"
Jul 24 01:36:52.859715 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.859238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882" event={"ID":"4dfc01fd-8856-4a6d-ac2a-141b0c2545f5","Type":"ContainerDied","Data":"fecdc3af657a0593399a55a94fe7d795ffda474009afea345661b09120bf91a8"}
Jul 24 01:36:52.859715 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.859397    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fecdc3af657a0593399a55a94fe7d795ffda474009afea345661b09120bf91a8"
Jul 24 01:36:52.861224 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.861202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s"
Jul 24 01:36:52.861348 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.861204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s" event={"ID":"fd8be8f8-4127-4b32-8a63-12fa11d6065d","Type":"ContainerDied","Data":"f169d753e66d51f43e7fd18badf3dac423339e08d25d228211ccc7a133417f8e"}
Jul 24 01:36:52.861412 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:36:52.861351    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f169d753e66d51f43e7fd18badf3dac423339e08d25d228211ccc7a133417f8e"
Jul 24 01:37:10.408037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.407941    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5v684"]
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408324    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="extract"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408337    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="extract"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408348    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="extract"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408354    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="extract"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408363    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408371    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408377    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="pull"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408382    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="pull"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408389    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408394    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408412    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="pull"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408416    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="pull"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408422    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="extract"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408427    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="extract"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408436    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408441    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408446    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408452    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="util"
Jul 24 01:37:10.408450 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408460    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="pull"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408464    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="pull"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408471    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="pull"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408476    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="pull"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408482    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="extract"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408487    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="extract"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408546    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf33f2fe-f4d5-4d39-9d6c-78307c1d284f" containerName="extract"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408554    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dfc01fd-8856-4a6d-ac2a-141b0c2545f5" containerName="extract"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408561    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8be8f8-4127-4b32-8a63-12fa11d6065d" containerName="extract"
Jul 24 01:37:10.409002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.408567    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc00c075-0552-47df-8578-a5996bde6f76" containerName="extract"
Jul 24 01:37:10.411602 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.411587    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:10.415819 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.415791    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 01:37:10.418166 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.418144    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 01:37:10.419381 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.419363    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mkl8f\""
Jul 24 01:37:10.480392 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.480358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5v684"]
Jul 24 01:37:10.526711 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.526671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjvh\" (UniqueName: \"kubernetes.io/projected/58bbc007-999c-4148-a2c5-13768ec27358-kube-api-access-gpjvh\") pod \"authorino-operator-75cc8d5cb7-5v684\" (UID: \"58bbc007-999c-4148-a2c5-13768ec27358\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:10.628070 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.628015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjvh\" (UniqueName: \"kubernetes.io/projected/58bbc007-999c-4148-a2c5-13768ec27358-kube-api-access-gpjvh\") pod \"authorino-operator-75cc8d5cb7-5v684\" (UID: \"58bbc007-999c-4148-a2c5-13768ec27358\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:10.641425 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.641391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjvh\" (UniqueName: \"kubernetes.io/projected/58bbc007-999c-4148-a2c5-13768ec27358-kube-api-access-gpjvh\") pod \"authorino-operator-75cc8d5cb7-5v684\" (UID: \"58bbc007-999c-4148-a2c5-13768ec27358\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:10.722946 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.722905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:10.881632 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.881605    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5v684"]
Jul 24 01:37:10.883994 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:37:10.883954    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bbc007_999c_4148_a2c5_13768ec27358.slice/crio-7cf164678ef329c6e1d88ebcbb7327344f8fd4f9768af7aa46b78860a2b5bf79 WatchSource:0}: Error finding container 7cf164678ef329c6e1d88ebcbb7327344f8fd4f9768af7aa46b78860a2b5bf79: Status 404 returned error can't find the container with id 7cf164678ef329c6e1d88ebcbb7327344f8fd4f9768af7aa46b78860a2b5bf79
Jul 24 01:37:10.968171 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:10.968136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684" event={"ID":"58bbc007-999c-4148-a2c5-13768ec27358","Type":"ContainerStarted","Data":"7cf164678ef329c6e1d88ebcbb7327344f8fd4f9768af7aa46b78860a2b5bf79"}
Jul 24 01:37:13.986530 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:13.986496    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684" event={"ID":"58bbc007-999c-4148-a2c5-13768ec27358","Type":"ContainerStarted","Data":"5ec68ba3b3529ec4d07e5dbf6dd8de752812f3dcc3db87cba6f40d16c27823e8"}
Jul 24 01:37:13.986930 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:13.986631    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:14.005439 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:14.005385    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684" podStartSLOduration=1.797453493 podStartE2EDuration="4.005369861s" totalImagesPullingTime="2.207916368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:37:14.003177678 +0000 UTC m=+454.425244531" watchObservedRunningTime="2026-07-24 01:37:14.005369861 +0000 UTC m=+454.427436697"
Jul 24 01:37:24.995141 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:24.995108    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5v684"
Jul 24 01:37:26.052271 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.052239    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"]
Jul 24 01:37:26.057743 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.057718    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.060956 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.060937    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zkr7f\""
Jul 24 01:37:26.072740 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.072705    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"]
Jul 24 01:37:26.174191 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.174148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24688cc9-7a27-45f0-bd05-864e596b0d7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.174359 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.174209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgv2c\" (UniqueName: \"kubernetes.io/projected/24688cc9-7a27-45f0-bd05-864e596b0d7f-kube-api-access-pgv2c\") pod \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.275268 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.275214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgv2c\" (UniqueName: \"kubernetes.io/projected/24688cc9-7a27-45f0-bd05-864e596b0d7f-kube-api-access-pgv2c\") pod \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.275458 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.275321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24688cc9-7a27-45f0-bd05-864e596b0d7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.275670 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.275653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24688cc9-7a27-45f0-bd05-864e596b0d7f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.286268 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.286233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgv2c\" (UniqueName: \"kubernetes.io/projected/24688cc9-7a27-45f0-bd05-864e596b0d7f-kube-api-access-pgv2c\") pod \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.370973 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.370888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.430669 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.430630    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"]
Jul 24 01:37:26.439932 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.439889    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"]
Jul 24 01:37:26.452256 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.452227    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"]
Jul 24 01:37:26.457893 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.457859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.471724 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.471695    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"]
Jul 24 01:37:26.578370 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.578338    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9096366c-5282-4d76-b6a6-0900f02d8abe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s9tfw\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.578501 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.578384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7m5c\" (UniqueName: \"kubernetes.io/projected/9096366c-5282-4d76-b6a6-0900f02d8abe-kube-api-access-l7m5c\") pod \"kuadrant-operator-controller-manager-55467d9f7-s9tfw\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.636876 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:26.636834    2564 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 24 01:37:26.636876 ip-10-0-129-127 kubenswrapper[2564]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-5ls2f_kuadrant-system_24688cc9-7a27-45f0-bd05-864e596b0d7f_0(774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-5ls2f to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0" Netns:"/var/run/netns/b09417c9-53cc-4e54-ab68-1afe7f4b65a6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-5ls2f;K8S_POD_INFRA_CONTAINER_ID=774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0;K8S_POD_UID=24688cc9-7a27-45f0-bd05-864e596b0d7f" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f/24688cc9-7a27-45f0-bd05-864e596b0d7f:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f 774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:29 [10.132.0.41/23]
Jul 24 01:37:26.636876 ip-10-0-129-127 kubenswrapper[2564]:         '
Jul 24 01:37:26.636876 ip-10-0-129-127 kubenswrapper[2564]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 24 01:37:26.636876 ip-10-0-129-127 kubenswrapper[2564]:  >
Jul 24 01:37:26.637090 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:26.636914    2564 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 24 01:37:26.637090 ip-10-0-129-127 kubenswrapper[2564]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-5ls2f_kuadrant-system_24688cc9-7a27-45f0-bd05-864e596b0d7f_0(774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-5ls2f to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0" Netns:"/var/run/netns/b09417c9-53cc-4e54-ab68-1afe7f4b65a6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-5ls2f;K8S_POD_INFRA_CONTAINER_ID=774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0;K8S_POD_UID=24688cc9-7a27-45f0-bd05-864e596b0d7f" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f] networking: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f/24688cc9-7a27-45f0-bd05-864e596b0d7f:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f 774ce094f6a0c7b9b4b099cd8fb107db25d6c4ec7e4063e63069a7802089d9f0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:29 [10.132.0.41/23]
Jul 24 01:37:26.637090 ip-10-0-129-127 kubenswrapper[2564]:         '
Jul 24 01:37:26.637090 ip-10-0-129-127 kubenswrapper[2564]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 24 01:37:26.637090 ip-10-0-129-127 kubenswrapper[2564]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:26.679797 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.679754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9096366c-5282-4d76-b6a6-0900f02d8abe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s9tfw\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.679975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.679801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7m5c\" (UniqueName: \"kubernetes.io/projected/9096366c-5282-4d76-b6a6-0900f02d8abe-kube-api-access-l7m5c\") pod \"kuadrant-operator-controller-manager-55467d9f7-s9tfw\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.680149 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.680127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9096366c-5282-4d76-b6a6-0900f02d8abe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-s9tfw\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.690077 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.690054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7m5c\" (UniqueName: \"kubernetes.io/projected/9096366c-5282-4d76-b6a6-0900f02d8abe-kube-api-access-l7m5c\") pod \"kuadrant-operator-controller-manager-55467d9f7-s9tfw\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.774928 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.774886    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:26.918857 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:26.918828    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"]
Jul 24 01:37:26.919523 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:37:26.919500    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9096366c_5282_4d76_b6a6_0900f02d8abe.slice/crio-54795599a5a498164041779060228c1576069d4d5ea9f303087a4d6066a73a1f WatchSource:0}: Error finding container 54795599a5a498164041779060228c1576069d4d5ea9f303087a4d6066a73a1f: Status 404 returned error can't find the container with id 54795599a5a498164041779060228c1576069d4d5ea9f303087a4d6066a73a1f
Jul 24 01:37:27.061453 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.061412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw" event={"ID":"9096366c-5282-4d76-b6a6-0900f02d8abe","Type":"ContainerStarted","Data":"54795599a5a498164041779060228c1576069d4d5ea9f303087a4d6066a73a1f"}
Jul 24 01:37:27.061832 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.061461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:27.063835 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:27.063804    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" is forbidden: User \"system:node:ip-10-0-129-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-127.ec2.internal' and this object" podUID="24688cc9-7a27-45f0-bd05-864e596b0d7f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:27.069406 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.069387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:27.071545 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:27.071519    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" is forbidden: User \"system:node:ip-10-0-129-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-127.ec2.internal' and this object" podUID="24688cc9-7a27-45f0-bd05-864e596b0d7f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:27.184800 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.184715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgv2c\" (UniqueName: \"kubernetes.io/projected/24688cc9-7a27-45f0-bd05-864e596b0d7f-kube-api-access-pgv2c\") pod \"24688cc9-7a27-45f0-bd05-864e596b0d7f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") "
Jul 24 01:37:27.184970 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.184858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24688cc9-7a27-45f0-bd05-864e596b0d7f-extensions-socket-volume\") pod \"24688cc9-7a27-45f0-bd05-864e596b0d7f\" (UID: \"24688cc9-7a27-45f0-bd05-864e596b0d7f\") "
Jul 24 01:37:27.185103 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.185079    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24688cc9-7a27-45f0-bd05-864e596b0d7f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "24688cc9-7a27-45f0-bd05-864e596b0d7f" (UID: "24688cc9-7a27-45f0-bd05-864e596b0d7f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:37:27.185161 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.185150    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24688cc9-7a27-45f0-bd05-864e596b0d7f-extensions-socket-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:37:27.187012 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.186989    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24688cc9-7a27-45f0-bd05-864e596b0d7f-kube-api-access-pgv2c" (OuterVolumeSpecName: "kube-api-access-pgv2c") pod "24688cc9-7a27-45f0-bd05-864e596b0d7f" (UID: "24688cc9-7a27-45f0-bd05-864e596b0d7f"). InnerVolumeSpecName "kube-api-access-pgv2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:37:27.286462 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:27.286427    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgv2c\" (UniqueName: \"kubernetes.io/projected/24688cc9-7a27-45f0-bd05-864e596b0d7f-kube-api-access-pgv2c\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:37:28.067687 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:28.067658    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:28.069995 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:28.069967    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" is forbidden: User \"system:node:ip-10-0-129-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-127.ec2.internal' and this object" podUID="24688cc9-7a27-45f0-bd05-864e596b0d7f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:28.075430 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:28.075400    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5ls2f\" is forbidden: User \"system:node:ip-10-0-129-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-127.ec2.internal' and this object" podUID="24688cc9-7a27-45f0-bd05-864e596b0d7f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5ls2f"
Jul 24 01:37:28.211369 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:28.211323    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24688cc9-7a27-45f0-bd05-864e596b0d7f" path="/var/lib/kubelet/pods/24688cc9-7a27-45f0-bd05-864e596b0d7f/volumes"
Jul 24 01:37:32.090911 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:32.090873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw" event={"ID":"9096366c-5282-4d76-b6a6-0900f02d8abe","Type":"ContainerStarted","Data":"fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916"}
Jul 24 01:37:32.091299 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:32.091011    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:43.098659 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:43.098620    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:43.118449 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:43.118386    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw" podStartSLOduration=12.586318378 podStartE2EDuration="17.118370589s" totalImagesPullingTime="4.532052211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:37:32.113266604 +0000 UTC m=+472.535333441" watchObservedRunningTime="2026-07-24 01:37:43.118370589 +0000 UTC m=+483.540437424"
Jul 24 01:37:55.774598 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:55.774558    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"]
Jul 24 01:37:55.775241 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:55.774850    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw" podUID="9096366c-5282-4d76-b6a6-0900f02d8abe" containerName="manager" containerID="cri-o://fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916" gracePeriod=10
Jul 24 01:37:56.033721 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.033657    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:56.130365 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.130331    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9096366c-5282-4d76-b6a6-0900f02d8abe-extensions-socket-volume\") pod \"9096366c-5282-4d76-b6a6-0900f02d8abe\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") "
Jul 24 01:37:56.130365 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.130373    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7m5c\" (UniqueName: \"kubernetes.io/projected/9096366c-5282-4d76-b6a6-0900f02d8abe-kube-api-access-l7m5c\") pod \"9096366c-5282-4d76-b6a6-0900f02d8abe\" (UID: \"9096366c-5282-4d76-b6a6-0900f02d8abe\") "
Jul 24 01:37:56.130753 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.130728    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9096366c-5282-4d76-b6a6-0900f02d8abe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9096366c-5282-4d76-b6a6-0900f02d8abe" (UID: "9096366c-5282-4d76-b6a6-0900f02d8abe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:37:56.132509 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.132483    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9096366c-5282-4d76-b6a6-0900f02d8abe-kube-api-access-l7m5c" (OuterVolumeSpecName: "kube-api-access-l7m5c") pod "9096366c-5282-4d76-b6a6-0900f02d8abe" (UID: "9096366c-5282-4d76-b6a6-0900f02d8abe"). InnerVolumeSpecName "kube-api-access-l7m5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:37:56.225037 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.224995    2564 generic.go:356] "Generic (PLEG): container finished" podID="9096366c-5282-4d76-b6a6-0900f02d8abe" containerID="fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916" exitCode=0
Jul 24 01:37:56.225179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.225063    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"
Jul 24 01:37:56.225179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.225083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw" event={"ID":"9096366c-5282-4d76-b6a6-0900f02d8abe","Type":"ContainerDied","Data":"fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916"}
Jul 24 01:37:56.225179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.225130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw" event={"ID":"9096366c-5282-4d76-b6a6-0900f02d8abe","Type":"ContainerDied","Data":"54795599a5a498164041779060228c1576069d4d5ea9f303087a4d6066a73a1f"}
Jul 24 01:37:56.225179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.225152    2564 scope.go:117] "RemoveContainer" containerID="fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916"
Jul 24 01:37:56.231504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.231483    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9096366c-5282-4d76-b6a6-0900f02d8abe-extensions-socket-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:37:56.231610 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.231514    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7m5c\" (UniqueName: \"kubernetes.io/projected/9096366c-5282-4d76-b6a6-0900f02d8abe-kube-api-access-l7m5c\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:37:56.235422 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.235403    2564 scope.go:117] "RemoveContainer" containerID="fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916"
Jul 24 01:37:56.235674 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:37:56.235654    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916\": container with ID starting with fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916 not found: ID does not exist" containerID="fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916"
Jul 24 01:37:56.235739 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.235686    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916"} err="failed to get container status \"fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916\": rpc error: code = NotFound desc = could not find container \"fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916\": container with ID starting with fa27e3ab4f5dafd63a12fe4342f11b98105b8932ffe3dcd459b206218b2c1916 not found: ID does not exist"
Jul 24 01:37:56.240944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.240920    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"]
Jul 24 01:37:56.246461 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:56.246433    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-s9tfw"]
Jul 24 01:37:58.210577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:58.210541    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9096366c-5282-4d76-b6a6-0900f02d8abe" path="/var/lib/kubelet/pods/9096366c-5282-4d76-b6a6-0900f02d8abe/volumes"
Jul 24 01:37:59.556963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.556934    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"]
Jul 24 01:37:59.557356 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.557339    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9096366c-5282-4d76-b6a6-0900f02d8abe" containerName="manager"
Jul 24 01:37:59.557356 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.557351    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9096366c-5282-4d76-b6a6-0900f02d8abe" containerName="manager"
Jul 24 01:37:59.557434 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.557409    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9096366c-5282-4d76-b6a6-0900f02d8abe" containerName="manager"
Jul 24 01:37:59.569883 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.569855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.571225 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.571197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"]
Jul 24 01:37:59.572186 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.572159    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l6t95\""
Jul 24 01:37:59.666929 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.666896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667104 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.666944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667104 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.666981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667104 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.667076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667253 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.667120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntb4p\" (UniqueName: \"kubernetes.io/projected/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-kube-api-access-ntb4p\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667253 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.667138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667253 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.667175    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667389 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.667246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.667389 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.667294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768301 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntb4p\" (UniqueName: \"kubernetes.io/projected/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-kube-api-access-ntb4p\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768511 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768985 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.768985 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.768979    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.769284 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.769262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.769378 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.769300    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.769378 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.769304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.770975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.770953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.771070 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.771049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.776862 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.776840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntb4p\" (UniqueName: \"kubernetes.io/projected/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-kube-api-access-ntb4p\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.776963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.776843    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc67a0c-d029-4f2b-afdc-72fb44fc7c17-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-g7hqg\" (UID: \"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:37:59.881641 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:37:59.881553    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:38:00.026539 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:00.026508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"]
Jul 24 01:38:00.027995 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:38:00.027968    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc67a0c_d029_4f2b_afdc_72fb44fc7c17.slice/crio-b709ec710724c4229560c279ba8d1ef7cfd2ba2a7d806d64952f4193a1b9553b WatchSource:0}: Error finding container b709ec710724c4229560c279ba8d1ef7cfd2ba2a7d806d64952f4193a1b9553b: Status 404 returned error can't find the container with id b709ec710724c4229560c279ba8d1ef7cfd2ba2a7d806d64952f4193a1b9553b
Jul 24 01:38:00.249629 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:00.249596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" event={"ID":"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17","Type":"ContainerStarted","Data":"b709ec710724c4229560c279ba8d1ef7cfd2ba2a7d806d64952f4193a1b9553b"}
Jul 24 01:38:03.983146 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:03.983110    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:38:03.983442 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:03.983181    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:38:03.983442 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:03.983211    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:38:04.274834 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:04.274740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" event={"ID":"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17","Type":"ContainerStarted","Data":"e9cd42098b6ca227cd0472bed2fc8e392908cf9b8bee1f198b27824f584ade63"}
Jul 24 01:38:04.295040 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:04.294982    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" podStartSLOduration=1.342400548 podStartE2EDuration="5.294965291s" totalImagesPullingTime="3.952564743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:38:04.292275167 +0000 UTC m=+504.714342022" watchObservedRunningTime="2026-07-24 01:38:04.294965291 +0000 UTC m=+504.717032126"
Jul 24 01:38:04.882545 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:04.882513    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:38:05.887496 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:05.887463    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:38:06.286050 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:06.285988    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:38:06.287129 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:06.287112    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:38:15.682730 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.682692    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:15.685237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.685220    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.687396 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.687369    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 01:38:15.687540 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.687447    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5dwhk\""
Jul 24 01:38:15.695135 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.695103    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:15.784945 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.784907    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:15.823109 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.823075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a830e3a-766b-4379-bb1a-7d5a7818a02c-config-file\") pod \"limitador-limitador-64c94fd64f-bf87q\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.823268 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.823132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpffm\" (UniqueName: \"kubernetes.io/projected/2a830e3a-766b-4379-bb1a-7d5a7818a02c-kube-api-access-gpffm\") pod \"limitador-limitador-64c94fd64f-bf87q\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.924137 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.924091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a830e3a-766b-4379-bb1a-7d5a7818a02c-config-file\") pod \"limitador-limitador-64c94fd64f-bf87q\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.924320 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.924168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpffm\" (UniqueName: \"kubernetes.io/projected/2a830e3a-766b-4379-bb1a-7d5a7818a02c-kube-api-access-gpffm\") pod \"limitador-limitador-64c94fd64f-bf87q\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.924756 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.924735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a830e3a-766b-4379-bb1a-7d5a7818a02c-config-file\") pod \"limitador-limitador-64c94fd64f-bf87q\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.931608 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.931585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpffm\" (UniqueName: \"kubernetes.io/projected/2a830e3a-766b-4379-bb1a-7d5a7818a02c-kube-api-access-gpffm\") pod \"limitador-limitador-64c94fd64f-bf87q\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:15.996796 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:15.996757    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:16.135527 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:16.135500    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:16.136885 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:38:16.136857    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a830e3a_766b_4379_bb1a_7d5a7818a02c.slice/crio-e2772f0111ab5fc28314f05cedab5a80c3e0b7092db8c07413b7826126602dd6 WatchSource:0}: Error finding container e2772f0111ab5fc28314f05cedab5a80c3e0b7092db8c07413b7826126602dd6: Status 404 returned error can't find the container with id e2772f0111ab5fc28314f05cedab5a80c3e0b7092db8c07413b7826126602dd6
Jul 24 01:38:16.347594 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:16.347516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q" event={"ID":"2a830e3a-766b-4379-bb1a-7d5a7818a02c","Type":"ContainerStarted","Data":"e2772f0111ab5fc28314f05cedab5a80c3e0b7092db8c07413b7826126602dd6"}
Jul 24 01:38:19.366813 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:19.366775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q" event={"ID":"2a830e3a-766b-4379-bb1a-7d5a7818a02c","Type":"ContainerStarted","Data":"72db5a76429298ec2bf4d84fb2a632cec14aec585f33e7eb8ff131cb0f0c15c2"}
Jul 24 01:38:19.367302 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:19.366885    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:19.384041 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:19.383981    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q" podStartSLOduration=1.76731038 podStartE2EDuration="4.383965371s" totalImagesPullingTime="2.616654991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:38:19.380864209 +0000 UTC m=+519.802931044" watchObservedRunningTime="2026-07-24 01:38:19.383965371 +0000 UTC m=+519.806032206"
Jul 24 01:38:30.374209 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:30.374177    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:30.937081 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:30.937044    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:30.937259 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:30.937236    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q" podUID="2a830e3a-766b-4379-bb1a-7d5a7818a02c" containerName="limitador" containerID="cri-o://72db5a76429298ec2bf4d84fb2a632cec14aec585f33e7eb8ff131cb0f0c15c2" gracePeriod=30
Jul 24 01:38:31.434304 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.434273    2564 generic.go:356] "Generic (PLEG): container finished" podID="2a830e3a-766b-4379-bb1a-7d5a7818a02c" containerID="72db5a76429298ec2bf4d84fb2a632cec14aec585f33e7eb8ff131cb0f0c15c2" exitCode=0
Jul 24 01:38:31.434717 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.434348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q" event={"ID":"2a830e3a-766b-4379-bb1a-7d5a7818a02c","Type":"ContainerDied","Data":"72db5a76429298ec2bf4d84fb2a632cec14aec585f33e7eb8ff131cb0f0c15c2"}
Jul 24 01:38:31.490059 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.490017    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:31.555644 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.555567    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a830e3a-766b-4379-bb1a-7d5a7818a02c-config-file\") pod \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") "
Jul 24 01:38:31.555644 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.555603    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpffm\" (UniqueName: \"kubernetes.io/projected/2a830e3a-766b-4379-bb1a-7d5a7818a02c-kube-api-access-gpffm\") pod \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\" (UID: \"2a830e3a-766b-4379-bb1a-7d5a7818a02c\") "
Jul 24 01:38:31.555973 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.555953    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a830e3a-766b-4379-bb1a-7d5a7818a02c-config-file" (OuterVolumeSpecName: "config-file") pod "2a830e3a-766b-4379-bb1a-7d5a7818a02c" (UID: "2a830e3a-766b-4379-bb1a-7d5a7818a02c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:38:31.557682 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.557661    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a830e3a-766b-4379-bb1a-7d5a7818a02c-kube-api-access-gpffm" (OuterVolumeSpecName: "kube-api-access-gpffm") pod "2a830e3a-766b-4379-bb1a-7d5a7818a02c" (UID: "2a830e3a-766b-4379-bb1a-7d5a7818a02c"). InnerVolumeSpecName "kube-api-access-gpffm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:38:31.656808 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.656761    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a830e3a-766b-4379-bb1a-7d5a7818a02c-config-file\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:38:31.656808 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:31.656799    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpffm\" (UniqueName: \"kubernetes.io/projected/2a830e3a-766b-4379-bb1a-7d5a7818a02c-kube-api-access-gpffm\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:38:32.441257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:32.441184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q" event={"ID":"2a830e3a-766b-4379-bb1a-7d5a7818a02c","Type":"ContainerDied","Data":"e2772f0111ab5fc28314f05cedab5a80c3e0b7092db8c07413b7826126602dd6"}
Jul 24 01:38:32.441257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:32.441203    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bf87q"
Jul 24 01:38:32.441257 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:32.441242    2564 scope.go:117] "RemoveContainer" containerID="72db5a76429298ec2bf4d84fb2a632cec14aec585f33e7eb8ff131cb0f0c15c2"
Jul 24 01:38:32.456817 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:32.456791    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:32.462244 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:32.462216    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bf87q"]
Jul 24 01:38:34.210186 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:34.210154    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a830e3a-766b-4379-bb1a-7d5a7818a02c" path="/var/lib/kubelet/pods/2a830e3a-766b-4379-bb1a-7d5a7818a02c/volumes"
Jul 24 01:38:37.547999 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.547968    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"]
Jul 24 01:38:37.548388 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.548375    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a830e3a-766b-4379-bb1a-7d5a7818a02c" containerName="limitador"
Jul 24 01:38:37.548388 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.548386    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a830e3a-766b-4379-bb1a-7d5a7818a02c" containerName="limitador"
Jul 24 01:38:37.548461 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.548455    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a830e3a-766b-4379-bb1a-7d5a7818a02c" containerName="limitador"
Jul 24 01:38:37.553831 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.553809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.557131 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.556885    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 01:38:37.557282 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.557131    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 01:38:37.557282 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.557152    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 01:38:37.557405 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.557392    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ph985\""
Jul 24 01:38:37.559573 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.559551    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"]
Jul 24 01:38:37.610287 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.610251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35029611-0208-4b7c-9057-0a068d54e3ce-data\") pod \"postgres-69fb4594cf-4pr2x\" (UID: \"35029611-0208-4b7c-9057-0a068d54e3ce\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.610430 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.610301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8s7\" (UniqueName: \"kubernetes.io/projected/35029611-0208-4b7c-9057-0a068d54e3ce-kube-api-access-ps8s7\") pod \"postgres-69fb4594cf-4pr2x\" (UID: \"35029611-0208-4b7c-9057-0a068d54e3ce\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.711491 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.711456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35029611-0208-4b7c-9057-0a068d54e3ce-data\") pod \"postgres-69fb4594cf-4pr2x\" (UID: \"35029611-0208-4b7c-9057-0a068d54e3ce\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.711694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.711516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8s7\" (UniqueName: \"kubernetes.io/projected/35029611-0208-4b7c-9057-0a068d54e3ce-kube-api-access-ps8s7\") pod \"postgres-69fb4594cf-4pr2x\" (UID: \"35029611-0208-4b7c-9057-0a068d54e3ce\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.711858 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.711839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35029611-0208-4b7c-9057-0a068d54e3ce-data\") pod \"postgres-69fb4594cf-4pr2x\" (UID: \"35029611-0208-4b7c-9057-0a068d54e3ce\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.718952 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.718928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8s7\" (UniqueName: \"kubernetes.io/projected/35029611-0208-4b7c-9057-0a068d54e3ce-kube-api-access-ps8s7\") pod \"postgres-69fb4594cf-4pr2x\" (UID: \"35029611-0208-4b7c-9057-0a068d54e3ce\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:37.866057 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:37.865953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:38.002898 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:38.002866    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"]
Jul 24 01:38:38.004069 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:38:38.004041    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35029611_0208_4b7c_9057_0a068d54e3ce.slice/crio-5c3732c7c69ca4697085feba9b50535a1a8bd662fd675ec6f9b117333e5ea4ac WatchSource:0}: Error finding container 5c3732c7c69ca4697085feba9b50535a1a8bd662fd675ec6f9b117333e5ea4ac: Status 404 returned error can't find the container with id 5c3732c7c69ca4697085feba9b50535a1a8bd662fd675ec6f9b117333e5ea4ac
Jul 24 01:38:38.476125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:38.476090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x" event={"ID":"35029611-0208-4b7c-9057-0a068d54e3ce","Type":"ContainerStarted","Data":"5c3732c7c69ca4697085feba9b50535a1a8bd662fd675ec6f9b117333e5ea4ac"}
Jul 24 01:38:43.204261 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:43.204230    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 01:38:43.512789 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:43.512693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x" event={"ID":"35029611-0208-4b7c-9057-0a068d54e3ce","Type":"ContainerStarted","Data":"8ce1b5ec6f8dffc0a91207bbd24aa2b4be797ff69b7816a78fc38bcaca4d0f04"}
Jul 24 01:38:43.512789 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:43.512774    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:43.529172 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:43.529113    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x" podStartSLOduration=1.332782747 podStartE2EDuration="6.529096466s" totalImagesPullingTime="5.196313719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:38:43.525818861 +0000 UTC m=+543.947885696" watchObservedRunningTime="2026-07-24 01:38:43.529096466 +0000 UTC m=+543.951163302"
Jul 24 01:38:49.547238 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:49.547209    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4pr2x"
Jul 24 01:38:57.120648 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.120613    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-ddb96bd5c-qpjvc"]
Jul 24 01:38:57.126333 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.126305    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.129782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.129752    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7m6j8\""
Jul 24 01:38:57.129782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.129744    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 01:38:57.129782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.129783    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 01:38:57.133527 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.133492    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-ddb96bd5c-qpjvc"]
Jul 24 01:38:57.200751 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.200715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846516c6-9a57-4729-bd9a-c647419a4f3b-webhook-cert\") pod \"maas-controller-ddb96bd5c-qpjvc\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") " pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.200940 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.200804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnsff\" (UniqueName: \"kubernetes.io/projected/846516c6-9a57-4729-bd9a-c647419a4f3b-kube-api-access-wnsff\") pod \"maas-controller-ddb96bd5c-qpjvc\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") " pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.301828 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.301793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846516c6-9a57-4729-bd9a-c647419a4f3b-webhook-cert\") pod \"maas-controller-ddb96bd5c-qpjvc\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") " pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.302018 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.301878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnsff\" (UniqueName: \"kubernetes.io/projected/846516c6-9a57-4729-bd9a-c647419a4f3b-kube-api-access-wnsff\") pod \"maas-controller-ddb96bd5c-qpjvc\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") " pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.304796 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.304769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846516c6-9a57-4729-bd9a-c647419a4f3b-webhook-cert\") pod \"maas-controller-ddb96bd5c-qpjvc\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") " pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.305379 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.305361    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-zkzqk"]
Jul 24 01:38:57.307995 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.307978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.310308 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.310282    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnsff\" (UniqueName: \"kubernetes.io/projected/846516c6-9a57-4729-bd9a-c647419a4f3b-kube-api-access-wnsff\") pod \"maas-controller-ddb96bd5c-qpjvc\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") " pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.317629 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.317601    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-zkzqk"]
Jul 24 01:38:57.402704 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.402622    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmlr\" (UniqueName: \"kubernetes.io/projected/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-kube-api-access-4dmlr\") pod \"maas-controller-5d9cdf4547-zkzqk\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") " pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.402850 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.402723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-webhook-cert\") pod \"maas-controller-5d9cdf4547-zkzqk\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") " pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.439112 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.439073    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:38:57.503773 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.503568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmlr\" (UniqueName: \"kubernetes.io/projected/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-kube-api-access-4dmlr\") pod \"maas-controller-5d9cdf4547-zkzqk\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") " pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.503773 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.503676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-webhook-cert\") pod \"maas-controller-5d9cdf4547-zkzqk\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") " pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.506549 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.506525    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-webhook-cert\") pod \"maas-controller-5d9cdf4547-zkzqk\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") " pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.512382 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.512354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmlr\" (UniqueName: \"kubernetes.io/projected/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-kube-api-access-4dmlr\") pod \"maas-controller-5d9cdf4547-zkzqk\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") " pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.575082 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.575057    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-ddb96bd5c-qpjvc"]
Jul 24 01:38:57.576654 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:38:57.576615    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846516c6_9a57_4729_bd9a_c647419a4f3b.slice/crio-b2991d46712f9e2149b2ef0bc2a3b7ca4004b43db1767bdaa18735259a265b57 WatchSource:0}: Error finding container b2991d46712f9e2149b2ef0bc2a3b7ca4004b43db1767bdaa18735259a265b57: Status 404 returned error can't find the container with id b2991d46712f9e2149b2ef0bc2a3b7ca4004b43db1767bdaa18735259a265b57
Jul 24 01:38:57.591683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.591657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc" event={"ID":"846516c6-9a57-4729-bd9a-c647419a4f3b","Type":"ContainerStarted","Data":"b2991d46712f9e2149b2ef0bc2a3b7ca4004b43db1767bdaa18735259a265b57"}
Jul 24 01:38:57.631057 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.631009    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:38:57.767182 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:57.767157    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-zkzqk"]
Jul 24 01:38:57.768806 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:38:57.768776    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd1f6e6_cc9a_4bf7_ae01_31dc5d70923a.slice/crio-aa94cf37677f1f38626cf2024ee197a4770d6a920c9f9cad4a37637c46591038 WatchSource:0}: Error finding container aa94cf37677f1f38626cf2024ee197a4770d6a920c9f9cad4a37637c46591038: Status 404 returned error can't find the container with id aa94cf37677f1f38626cf2024ee197a4770d6a920c9f9cad4a37637c46591038
Jul 24 01:38:58.603112 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:38:58.603045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk" event={"ID":"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a","Type":"ContainerStarted","Data":"aa94cf37677f1f38626cf2024ee197a4770d6a920c9f9cad4a37637c46591038"}
Jul 24 01:39:00.618393 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:00.618345    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk" event={"ID":"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a","Type":"ContainerStarted","Data":"9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52"}
Jul 24 01:39:00.618878 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:00.618411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:39:00.619915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:00.619885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc" event={"ID":"846516c6-9a57-4729-bd9a-c647419a4f3b","Type":"ContainerStarted","Data":"5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42"}
Jul 24 01:39:00.620002 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:00.619942    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:39:00.641717 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:00.641667    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk" podStartSLOduration=1.29398893 podStartE2EDuration="3.641650495s" totalImagesPullingTime="2.347661565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:39:00.639698348 +0000 UTC m=+561.061765203" watchObservedRunningTime="2026-07-24 01:39:00.641650495 +0000 UTC m=+561.063717378"
Jul 24 01:39:00.661732 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:00.661671    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc" podStartSLOduration=1.116828429 podStartE2EDuration="3.661652279s" totalImagesPullingTime="2.54482385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:39:00.658021357 +0000 UTC m=+561.080088214" watchObservedRunningTime="2026-07-24 01:39:00.661652279 +0000 UTC m=+561.083719115"
Jul 24 01:39:11.628710 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:11.628675    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:39:11.629179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:11.628754    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:39:11.689417 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:11.689384    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-ddb96bd5c-qpjvc"]
Jul 24 01:39:11.689622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:11.689581    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc" podUID="846516c6-9a57-4729-bd9a-c647419a4f3b" containerName="manager" containerID="cri-o://5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42" gracePeriod=30
Jul 24 01:39:11.946083 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:11.946053    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:39:12.040409 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.040377    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846516c6-9a57-4729-bd9a-c647419a4f3b-webhook-cert\") pod \"846516c6-9a57-4729-bd9a-c647419a4f3b\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") "
Jul 24 01:39:12.040569 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.040448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnsff\" (UniqueName: \"kubernetes.io/projected/846516c6-9a57-4729-bd9a-c647419a4f3b-kube-api-access-wnsff\") pod \"846516c6-9a57-4729-bd9a-c647419a4f3b\" (UID: \"846516c6-9a57-4729-bd9a-c647419a4f3b\") "
Jul 24 01:39:12.042842 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.042812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846516c6-9a57-4729-bd9a-c647419a4f3b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "846516c6-9a57-4729-bd9a-c647419a4f3b" (UID: "846516c6-9a57-4729-bd9a-c647419a4f3b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:39:12.042995 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.042884    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846516c6-9a57-4729-bd9a-c647419a4f3b-kube-api-access-wnsff" (OuterVolumeSpecName: "kube-api-access-wnsff") pod "846516c6-9a57-4729-bd9a-c647419a4f3b" (UID: "846516c6-9a57-4729-bd9a-c647419a4f3b"). InnerVolumeSpecName "kube-api-access-wnsff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:39:12.141139 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.141105    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnsff\" (UniqueName: \"kubernetes.io/projected/846516c6-9a57-4729-bd9a-c647419a4f3b-kube-api-access-wnsff\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:39:12.141139 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.141136    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846516c6-9a57-4729-bd9a-c647419a4f3b-webhook-cert\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:39:12.691077 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.690984    2564 generic.go:356] "Generic (PLEG): container finished" podID="846516c6-9a57-4729-bd9a-c647419a4f3b" containerID="5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42" exitCode=0
Jul 24 01:39:12.691077 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.691048    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc"
Jul 24 01:39:12.691542 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.691075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc" event={"ID":"846516c6-9a57-4729-bd9a-c647419a4f3b","Type":"ContainerDied","Data":"5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42"}
Jul 24 01:39:12.691542 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.691123    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ddb96bd5c-qpjvc" event={"ID":"846516c6-9a57-4729-bd9a-c647419a4f3b","Type":"ContainerDied","Data":"b2991d46712f9e2149b2ef0bc2a3b7ca4004b43db1767bdaa18735259a265b57"}
Jul 24 01:39:12.691542 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.691145    2564 scope.go:117] "RemoveContainer" containerID="5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42"
Jul 24 01:39:12.700886 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.700865    2564 scope.go:117] "RemoveContainer" containerID="5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42"
Jul 24 01:39:12.701170 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:39:12.701148    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42\": container with ID starting with 5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42 not found: ID does not exist" containerID="5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42"
Jul 24 01:39:12.701225 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.701180    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42"} err="failed to get container status \"5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42\": rpc error: code = NotFound desc = could not find container \"5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42\": container with ID starting with 5ffbe29359df83d2d67eab549a32b3f4edffe0fa00085c39932797ce922ede42 not found: ID does not exist"
Jul 24 01:39:12.707483 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.707458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-ddb96bd5c-qpjvc"]
Jul 24 01:39:12.713666 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:12.713644    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-ddb96bd5c-qpjvc"]
Jul 24 01:39:14.210138 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:14.210106    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846516c6-9a57-4729-bd9a-c647419a4f3b" path="/var/lib/kubelet/pods/846516c6-9a57-4729-bd9a-c647419a4f3b/volumes"
Jul 24 01:39:31.087459 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.087422    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"]
Jul 24 01:39:31.087899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.087825    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846516c6-9a57-4729-bd9a-c647419a4f3b" containerName="manager"
Jul 24 01:39:31.087899 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.087835    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="846516c6-9a57-4729-bd9a-c647419a4f3b" containerName="manager"
Jul 24 01:39:31.087979 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.087903    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="846516c6-9a57-4729-bd9a-c647419a4f3b" containerName="manager"
Jul 24 01:39:31.116412 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.116377    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"]
Jul 24 01:39:31.116565 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.116510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.118611 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.118590    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 01:39:31.118761 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.118743    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-76qzn\""
Jul 24 01:39:31.211577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.211534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80ff20c-12e1-4bd1-a14d-f50c734e0756-openshift-service-ca\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.211772 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.211624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80ff20c-12e1-4bd1-a14d-f50c734e0756-maas-api-tls\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.211772 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.211688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcft7\" (UniqueName: \"kubernetes.io/projected/a80ff20c-12e1-4bd1-a14d-f50c734e0756-kube-api-access-tcft7\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.313179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.312991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80ff20c-12e1-4bd1-a14d-f50c734e0756-maas-api-tls\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.313179 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.313079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcft7\" (UniqueName: \"kubernetes.io/projected/a80ff20c-12e1-4bd1-a14d-f50c734e0756-kube-api-access-tcft7\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.313460 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.313256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80ff20c-12e1-4bd1-a14d-f50c734e0756-openshift-service-ca\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.313939 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.313918    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80ff20c-12e1-4bd1-a14d-f50c734e0756-openshift-service-ca\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.315645 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.315620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80ff20c-12e1-4bd1-a14d-f50c734e0756-maas-api-tls\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.320915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.320890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcft7\" (UniqueName: \"kubernetes.io/projected/a80ff20c-12e1-4bd1-a14d-f50c734e0756-kube-api-access-tcft7\") pod \"maas-api-67d795fcc8-sc4gd\" (UID: \"a80ff20c-12e1-4bd1-a14d-f50c734e0756\") " pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.427426 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.427337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:31.576505 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:39:31.576474    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80ff20c_12e1_4bd1_a14d_f50c734e0756.slice/crio-24c6b4e82cc2b28ce7e29a0d6c4281317b434ef46cc59c9b03992c1790253b2f WatchSource:0}: Error finding container 24c6b4e82cc2b28ce7e29a0d6c4281317b434ef46cc59c9b03992c1790253b2f: Status 404 returned error can't find the container with id 24c6b4e82cc2b28ce7e29a0d6c4281317b434ef46cc59c9b03992c1790253b2f
Jul 24 01:39:31.576657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.576622    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"]
Jul 24 01:39:31.753874 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.753839    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tmzhk"]
Jul 24 01:39:31.758656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.758630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:31.761011 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.760987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 01:39:31.761159 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.761131    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kj9c5\""
Jul 24 01:39:31.768162 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.768138    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tmzhk"]
Jul 24 01:39:31.804354 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.804320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd" event={"ID":"a80ff20c-12e1-4bd1-a14d-f50c734e0756","Type":"ContainerStarted","Data":"24c6b4e82cc2b28ce7e29a0d6c4281317b434ef46cc59c9b03992c1790253b2f"}
Jul 24 01:39:31.817810 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.817784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639e4494-3463-4707-9680-2a2ccc9d555c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tmzhk\" (UID: \"639e4494-3463-4707-9680-2a2ccc9d555c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:31.817964 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.817902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94p7j\" (UniqueName: \"kubernetes.io/projected/639e4494-3463-4707-9680-2a2ccc9d555c-kube-api-access-94p7j\") pod \"payload-processing-5cc6cd444b-tmzhk\" (UID: \"639e4494-3463-4707-9680-2a2ccc9d555c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:31.919307 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.919268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639e4494-3463-4707-9680-2a2ccc9d555c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tmzhk\" (UID: \"639e4494-3463-4707-9680-2a2ccc9d555c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:31.919499 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.919451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94p7j\" (UniqueName: \"kubernetes.io/projected/639e4494-3463-4707-9680-2a2ccc9d555c-kube-api-access-94p7j\") pod \"payload-processing-5cc6cd444b-tmzhk\" (UID: \"639e4494-3463-4707-9680-2a2ccc9d555c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:31.920220 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.920194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639e4494-3463-4707-9680-2a2ccc9d555c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tmzhk\" (UID: \"639e4494-3463-4707-9680-2a2ccc9d555c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:31.926959 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:31.926922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94p7j\" (UniqueName: \"kubernetes.io/projected/639e4494-3463-4707-9680-2a2ccc9d555c-kube-api-access-94p7j\") pod \"payload-processing-5cc6cd444b-tmzhk\" (UID: \"639e4494-3463-4707-9680-2a2ccc9d555c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:32.071868 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:32.071766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:32.438104 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:32.437478    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tmzhk"]
Jul 24 01:39:32.446110 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:39:32.446075    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639e4494_3463_4707_9680_2a2ccc9d555c.slice/crio-29194f9f78b049fecd06dd4e11927df818e8b610ad0f78d70816350ba0123e21 WatchSource:0}: Error finding container 29194f9f78b049fecd06dd4e11927df818e8b610ad0f78d70816350ba0123e21: Status 404 returned error can't find the container with id 29194f9f78b049fecd06dd4e11927df818e8b610ad0f78d70816350ba0123e21
Jul 24 01:39:32.834691 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:32.832153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk" event={"ID":"639e4494-3463-4707-9680-2a2ccc9d555c","Type":"ContainerStarted","Data":"29194f9f78b049fecd06dd4e11927df818e8b610ad0f78d70816350ba0123e21"}
Jul 24 01:39:33.851279 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:33.851232    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd" event={"ID":"a80ff20c-12e1-4bd1-a14d-f50c734e0756","Type":"ContainerStarted","Data":"6cdf5c60fdb6824593d34caafe8fab3c7705859fea86752cf67779ae48b8765c"}
Jul 24 01:39:33.851669 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:33.851430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:33.869947 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:33.869285    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd" podStartSLOduration=0.750433835 podStartE2EDuration="2.869267318s" totalImagesPullingTime="2.118833483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:39:33.866464506 +0000 UTC m=+594.288531366" watchObservedRunningTime="2026-07-24 01:39:33.869267318 +0000 UTC m=+594.291334154"
Jul 24 01:39:34.859244 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:34.859205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk" event={"ID":"639e4494-3463-4707-9680-2a2ccc9d555c","Type":"ContainerStarted","Data":"5d6613c96dd676e0e821b466275296a72dcd494ac6e0b6f9b73c7e768cd57603"}
Jul 24 01:39:34.859683 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:34.859296    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:34.875161 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:34.875103    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk" podStartSLOduration=1.9782633299999999 podStartE2EDuration="3.875086564s" totalImagesPullingTime="1.896823234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:39:34.873395322 +0000 UTC m=+595.295462201" watchObservedRunningTime="2026-07-24 01:39:34.875086564 +0000 UTC m=+595.297153398"
Jul 24 01:39:39.865625 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.865590    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67d795fcc8-sc4gd"
Jul 24 01:39:39.894488 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.894441    2564 generic.go:356] "Generic (PLEG): container finished" podID="3fc67a0c-d029-4f2b-afdc-72fb44fc7c17" containerID="e9cd42098b6ca227cd0472bed2fc8e392908cf9b8bee1f198b27824f584ade63" exitCode=0
Jul 24 01:39:39.894657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.894492    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" event={"ID":"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17","Type":"ContainerDied","Data":"e9cd42098b6ca227cd0472bed2fc8e392908cf9b8bee1f198b27824f584ade63"}
Jul 24 01:39:39.894975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.894956    2564 scope.go:117] "RemoveContainer" containerID="e9cd42098b6ca227cd0472bed2fc8e392908cf9b8bee1f198b27824f584ade63"
Jul 24 01:39:39.896071 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.896041    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:39:39.896181 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.896127    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:39:39.896244 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:39.896181    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:39:40.326586 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:40.326555    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:39:40.327807 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:40.327782    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:39:40.334816 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:40.334793    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:39:40.335655 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:40.335639    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:39:40.902164 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:40.902125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" event={"ID":"3fc67a0c-d029-4f2b-afdc-72fb44fc7c17","Type":"ContainerStarted","Data":"696318ae1522415bd5f0717b1d07db25b873609db49d878513130bc50b12b464"}
Jul 24 01:39:41.882567 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:41.882533    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:39:41.882749 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:41.882727    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-g7hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 24 01:39:41.882794 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:41.882777    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" podUID="3fc67a0c-d029-4f2b-afdc-72fb44fc7c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 24 01:39:42.882674 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:42.882634    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-g7hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 24 01:39:42.883108 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:42.882693    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" podUID="3fc67a0c-d029-4f2b-afdc-72fb44fc7c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 24 01:39:43.882055 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:43.881992    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-g7hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 24 01:39:43.882235 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:43.882077    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg" podUID="3fc67a0c-d029-4f2b-afdc-72fb44fc7c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 24 01:39:44.910853 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:44.910810    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:39:44.911273 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:44.911123    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:39:44.911888 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:44.911869    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-g7hqg"
Jul 24 01:39:45.867727 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:45.867697    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-tmzhk"
Jul 24 01:39:50.159741 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.159709    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"]
Jul 24 01:39:50.163419 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.163393    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.165714 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.165683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bqxvq\""
Jul 24 01:39:50.165838 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.165688    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 01:39:50.166507 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.166489    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 01:39:50.166670 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.166650    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 24 01:39:50.174088 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.174066    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"]
Jul 24 01:39:50.313641 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.313589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.313641 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.313630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.313898 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.313798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c18e68-73fe-4683-8910-9d0126fb5bcf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.313963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.313898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.313963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.313936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqqm\" (UniqueName: \"kubernetes.io/projected/19c18e68-73fe-4683-8910-9d0126fb5bcf-kube-api-access-5kqqm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.314113 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.313972    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.414680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.414569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.414680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.414632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqqm\" (UniqueName: \"kubernetes.io/projected/19c18e68-73fe-4683-8910-9d0126fb5bcf-kube-api-access-5kqqm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.414680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.414674    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.414989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.414707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.414989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.414728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.414989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.414878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c18e68-73fe-4683-8910-9d0126fb5bcf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.415202 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.415156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.415301 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.415215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.415301 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.415248    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.417007 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.416978    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19c18e68-73fe-4683-8910-9d0126fb5bcf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.417328 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.417309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c18e68-73fe-4683-8910-9d0126fb5bcf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.421953 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.421930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqqm\" (UniqueName: \"kubernetes.io/projected/19c18e68-73fe-4683-8910-9d0126fb5bcf-kube-api-access-5kqqm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r\" (UID: \"19c18e68-73fe-4683-8910-9d0126fb5bcf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.475897 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.475861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:39:50.612182 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.612146    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"]
Jul 24 01:39:50.613890 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:39:50.613851    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c18e68_73fe_4683_8910_9d0126fb5bcf.slice/crio-3ee9991ff77990224e6422fbaa815b47ab913f6ef46c9aefa7684cf7fd14a005 WatchSource:0}: Error finding container 3ee9991ff77990224e6422fbaa815b47ab913f6ef46c9aefa7684cf7fd14a005: Status 404 returned error can't find the container with id 3ee9991ff77990224e6422fbaa815b47ab913f6ef46c9aefa7684cf7fd14a005
Jul 24 01:39:50.990874 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:50.990835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r" event={"ID":"19c18e68-73fe-4683-8910-9d0126fb5bcf","Type":"ContainerStarted","Data":"3ee9991ff77990224e6422fbaa815b47ab913f6ef46c9aefa7684cf7fd14a005"}
Jul 24 01:39:51.998329 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:51.998238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r" event={"ID":"19c18e68-73fe-4683-8910-9d0126fb5bcf","Type":"ContainerStarted","Data":"565ddcecf5acda6855b7b273a9432057fa1804f68fd11ad2a0f78423e04feda4"}
Jul 24 01:39:52.014208 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:39:52.014158    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r" podStartSLOduration=0.931491468 podStartE2EDuration="2.014140038s" totalImagesPullingTime="1.08264857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:39:52.013635249 +0000 UTC m=+612.435702114" watchObservedRunningTime="2026-07-24 01:39:52.014140038 +0000 UTC m=+612.436206871"
Jul 24 01:40:00.476854 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:40:00.476818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:40:00.477354 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:40:00.476962    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:40:00.489427 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:40:00.489399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:40:01.060548 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:40:01.060515    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r"
Jul 24 01:42:33.364870 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.364837    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-zkzqk"]
Jul 24 01:42:33.365388 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.365062    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk" podUID="ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" containerName="manager" containerID="cri-o://9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52" gracePeriod=30
Jul 24 01:42:33.624656 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.624591    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:42:33.738286 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.738249    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-webhook-cert\") pod \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") "
Jul 24 01:42:33.738286 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.738292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dmlr\" (UniqueName: \"kubernetes.io/projected/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-kube-api-access-4dmlr\") pod \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\" (UID: \"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a\") "
Jul 24 01:42:33.740378 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.740344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" (UID: "ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:42:33.740520 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.740380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-kube-api-access-4dmlr" (OuterVolumeSpecName: "kube-api-access-4dmlr") pod "ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" (UID: "ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a"). InnerVolumeSpecName "kube-api-access-4dmlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:42:33.839700 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.839661    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-webhook-cert\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:42:33.839700 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.839691    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dmlr\" (UniqueName: \"kubernetes.io/projected/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a-kube-api-access-4dmlr\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:42:33.917517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.917419    2564 generic.go:356] "Generic (PLEG): container finished" podID="ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" containerID="9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52" exitCode=0
Jul 24 01:42:33.917517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.917466    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk"
Jul 24 01:42:33.917517 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.917501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk" event={"ID":"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a","Type":"ContainerDied","Data":"9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52"}
Jul 24 01:42:33.917745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.917562    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-zkzqk" event={"ID":"ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a","Type":"ContainerDied","Data":"aa94cf37677f1f38626cf2024ee197a4770d6a920c9f9cad4a37637c46591038"}
Jul 24 01:42:33.917745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.917594    2564 scope.go:117] "RemoveContainer" containerID="9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52"
Jul 24 01:42:33.927717 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.927698    2564 scope.go:117] "RemoveContainer" containerID="9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52"
Jul 24 01:42:33.927984 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:42:33.927962    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52\": container with ID starting with 9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52 not found: ID does not exist" containerID="9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52"
Jul 24 01:42:33.928064 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.927994    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52"} err="failed to get container status \"9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52\": rpc error: code = NotFound desc = could not find container \"9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52\": container with ID starting with 9cf22ea68545dee9764ac170eb4805ec947d207965a978cd36c3704e9ea24f52 not found: ID does not exist"
Jul 24 01:42:33.938659 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.938625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-zkzqk"]
Jul 24 01:42:33.940958 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:33.940937    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-zkzqk"]
Jul 24 01:42:34.212040 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:42:34.211985    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" path="/var/lib/kubelet/pods/ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a/volumes"
Jul 24 01:43:34.783596 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.783552    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-hkq45"]
Jul 24 01:43:34.784310 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.784289    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" containerName="manager"
Jul 24 01:43:34.784401 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.784312    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" containerName="manager"
Jul 24 01:43:34.784456 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.784432    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd1f6e6-cc9a-4bf7-ae01-31dc5d70923a" containerName="manager"
Jul 24 01:43:34.787539 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.787517    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:34.790600 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.790574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 01:43:34.790725 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.790576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7m6j8\""
Jul 24 01:43:34.790725 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.790587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 01:43:34.798377 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.798347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-hkq45"]
Jul 24 01:43:34.914481 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.914446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm747\" (UniqueName: \"kubernetes.io/projected/3e112fb2-15ce-4068-b761-ef4d15fca42e-kube-api-access-rm747\") pod \"maas-controller-5d9cdf4547-hkq45\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") " pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:34.914663 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:34.914492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e112fb2-15ce-4068-b761-ef4d15fca42e-webhook-cert\") pod \"maas-controller-5d9cdf4547-hkq45\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") " pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:35.015871 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.015831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm747\" (UniqueName: \"kubernetes.io/projected/3e112fb2-15ce-4068-b761-ef4d15fca42e-kube-api-access-rm747\") pod \"maas-controller-5d9cdf4547-hkq45\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") " pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:35.016093 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.015884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e112fb2-15ce-4068-b761-ef4d15fca42e-webhook-cert\") pod \"maas-controller-5d9cdf4547-hkq45\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") " pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:35.018366 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.018347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e112fb2-15ce-4068-b761-ef4d15fca42e-webhook-cert\") pod \"maas-controller-5d9cdf4547-hkq45\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") " pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:35.024080 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.024062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm747\" (UniqueName: \"kubernetes.io/projected/3e112fb2-15ce-4068-b761-ef4d15fca42e-kube-api-access-rm747\") pod \"maas-controller-5d9cdf4547-hkq45\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") " pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:35.099462 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.099377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:35.236416 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.236392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-hkq45"]
Jul 24 01:43:35.237956 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:43:35.237919    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e112fb2_15ce_4068_b761_ef4d15fca42e.slice/crio-8684d630c159b4bbe3cafb77e800f31124e09a4be792e3438f3987de7d95f9cd WatchSource:0}: Error finding container 8684d630c159b4bbe3cafb77e800f31124e09a4be792e3438f3987de7d95f9cd: Status 404 returned error can't find the container with id 8684d630c159b4bbe3cafb77e800f31124e09a4be792e3438f3987de7d95f9cd
Jul 24 01:43:35.239369 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.239352    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 01:43:35.288879 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:35.288847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-hkq45" event={"ID":"3e112fb2-15ce-4068-b761-ef4d15fca42e","Type":"ContainerStarted","Data":"8684d630c159b4bbe3cafb77e800f31124e09a4be792e3438f3987de7d95f9cd"}
Jul 24 01:43:36.296233 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:36.296195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-hkq45" event={"ID":"3e112fb2-15ce-4068-b761-ef4d15fca42e","Type":"ContainerStarted","Data":"c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8"}
Jul 24 01:43:36.296615 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:36.296256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:36.312675 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:36.312629    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d9cdf4547-hkq45" podStartSLOduration=1.698746205 podStartE2EDuration="2.312613216s" totalImagesPullingTime="613.867011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:43:36.31120679 +0000 UTC m=+836.733273643" watchObservedRunningTime="2026-07-24 01:43:36.312613216 +0000 UTC m=+836.734680077"
Jul 24 01:43:47.304834 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:47.304788    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:51.981137 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:51.981087    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-hkq45"]
Jul 24 01:43:51.981626 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:51.981404    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d9cdf4547-hkq45" podUID="3e112fb2-15ce-4068-b761-ef4d15fca42e" containerName="manager" containerID="cri-o://c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8" gracePeriod=30
Jul 24 01:43:52.352833 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.352797    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:52.392214 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.392171    2564 generic.go:356] "Generic (PLEG): container finished" podID="3e112fb2-15ce-4068-b761-ef4d15fca42e" containerID="c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8" exitCode=0
Jul 24 01:43:52.392411 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.392231    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-hkq45"
Jul 24 01:43:52.392411 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.392344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-hkq45" event={"ID":"3e112fb2-15ce-4068-b761-ef4d15fca42e","Type":"ContainerDied","Data":"c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8"}
Jul 24 01:43:52.392536 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.392433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-hkq45" event={"ID":"3e112fb2-15ce-4068-b761-ef4d15fca42e","Type":"ContainerDied","Data":"8684d630c159b4bbe3cafb77e800f31124e09a4be792e3438f3987de7d95f9cd"}
Jul 24 01:43:52.392536 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.392470    2564 scope.go:117] "RemoveContainer" containerID="c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8"
Jul 24 01:43:52.413624 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.413603    2564 scope.go:117] "RemoveContainer" containerID="c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8"
Jul 24 01:43:52.413996 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:43:52.413972    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8\": container with ID starting with c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8 not found: ID does not exist" containerID="c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8"
Jul 24 01:43:52.414280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.414007    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8"} err="failed to get container status \"c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8\": rpc error: code = NotFound desc = could not find container \"c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8\": container with ID starting with c4b7dac5dae0f7fe0a467f24e0df543ff4f658518981e0659ccb34e2e40279b8 not found: ID does not exist"
Jul 24 01:43:52.491830 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.491795    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e112fb2-15ce-4068-b761-ef4d15fca42e-webhook-cert\") pod \"3e112fb2-15ce-4068-b761-ef4d15fca42e\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") "
Jul 24 01:43:52.492086 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.491842    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm747\" (UniqueName: \"kubernetes.io/projected/3e112fb2-15ce-4068-b761-ef4d15fca42e-kube-api-access-rm747\") pod \"3e112fb2-15ce-4068-b761-ef4d15fca42e\" (UID: \"3e112fb2-15ce-4068-b761-ef4d15fca42e\") "
Jul 24 01:43:52.494094 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.494055    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e112fb2-15ce-4068-b761-ef4d15fca42e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3e112fb2-15ce-4068-b761-ef4d15fca42e" (UID: "3e112fb2-15ce-4068-b761-ef4d15fca42e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:43:52.494204 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.494106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e112fb2-15ce-4068-b761-ef4d15fca42e-kube-api-access-rm747" (OuterVolumeSpecName: "kube-api-access-rm747") pod "3e112fb2-15ce-4068-b761-ef4d15fca42e" (UID: "3e112fb2-15ce-4068-b761-ef4d15fca42e"). InnerVolumeSpecName "kube-api-access-rm747". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:43:52.593473 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.593378    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e112fb2-15ce-4068-b761-ef4d15fca42e-webhook-cert\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:43:52.593473 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.593415    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm747\" (UniqueName: \"kubernetes.io/projected/3e112fb2-15ce-4068-b761-ef4d15fca42e-kube-api-access-rm747\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:43:52.716883 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.716848    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-hkq45"]
Jul 24 01:43:52.721660 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:52.721637    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-hkq45"]
Jul 24 01:43:53.552730 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.552696    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-tglhm"]
Jul 24 01:43:53.553296 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.553275    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e112fb2-15ce-4068-b761-ef4d15fca42e" containerName="manager"
Jul 24 01:43:53.553296 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.553295    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e112fb2-15ce-4068-b761-ef4d15fca42e" containerName="manager"
Jul 24 01:43:53.553412 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.553403    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e112fb2-15ce-4068-b761-ef4d15fca42e" containerName="manager"
Jul 24 01:43:53.557870 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.557854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.561464 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.561443    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 01:43:53.561464 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.561453    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7m6j8\""
Jul 24 01:43:53.561593 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.561445    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 01:43:53.567672 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.567645    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-tglhm"]
Jul 24 01:43:53.701602 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.701557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ndk\" (UniqueName: \"kubernetes.io/projected/55c23306-95d0-40e3-99eb-82bb0b50ff48-kube-api-access-64ndk\") pod \"maas-controller-5d9cdf4547-tglhm\" (UID: \"55c23306-95d0-40e3-99eb-82bb0b50ff48\") " pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.701792 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.701639    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c23306-95d0-40e3-99eb-82bb0b50ff48-webhook-cert\") pod \"maas-controller-5d9cdf4547-tglhm\" (UID: \"55c23306-95d0-40e3-99eb-82bb0b50ff48\") " pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.802985 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.802891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c23306-95d0-40e3-99eb-82bb0b50ff48-webhook-cert\") pod \"maas-controller-5d9cdf4547-tglhm\" (UID: \"55c23306-95d0-40e3-99eb-82bb0b50ff48\") " pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.803200 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.802987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ndk\" (UniqueName: \"kubernetes.io/projected/55c23306-95d0-40e3-99eb-82bb0b50ff48-kube-api-access-64ndk\") pod \"maas-controller-5d9cdf4547-tglhm\" (UID: \"55c23306-95d0-40e3-99eb-82bb0b50ff48\") " pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.805459 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.805433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55c23306-95d0-40e3-99eb-82bb0b50ff48-webhook-cert\") pod \"maas-controller-5d9cdf4547-tglhm\" (UID: \"55c23306-95d0-40e3-99eb-82bb0b50ff48\") " pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.811020 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.810993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ndk\" (UniqueName: \"kubernetes.io/projected/55c23306-95d0-40e3-99eb-82bb0b50ff48-kube-api-access-64ndk\") pod \"maas-controller-5d9cdf4547-tglhm\" (UID: \"55c23306-95d0-40e3-99eb-82bb0b50ff48\") " pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:53.870510 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:53.870468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:54.007313 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:54.007278    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d9cdf4547-tglhm"]
Jul 24 01:43:54.009142 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:43:54.009111    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c23306_95d0_40e3_99eb_82bb0b50ff48.slice/crio-d59f9b6c7ec04a223e0b2b399f6b6ea3866536410548b01f9ad66087591691ad WatchSource:0}: Error finding container d59f9b6c7ec04a223e0b2b399f6b6ea3866536410548b01f9ad66087591691ad: Status 404 returned error can't find the container with id d59f9b6c7ec04a223e0b2b399f6b6ea3866536410548b01f9ad66087591691ad
Jul 24 01:43:54.210372 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:54.210337    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e112fb2-15ce-4068-b761-ef4d15fca42e" path="/var/lib/kubelet/pods/3e112fb2-15ce-4068-b761-ef4d15fca42e/volumes"
Jul 24 01:43:54.406767 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:54.406727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-tglhm" event={"ID":"55c23306-95d0-40e3-99eb-82bb0b50ff48","Type":"ContainerStarted","Data":"d59f9b6c7ec04a223e0b2b399f6b6ea3866536410548b01f9ad66087591691ad"}
Jul 24 01:43:55.414001 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:55.413960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d9cdf4547-tglhm" event={"ID":"55c23306-95d0-40e3-99eb-82bb0b50ff48","Type":"ContainerStarted","Data":"e05b29697224a417536eee8a39a22db0fe2071bee56192224f081c22dff06fcf"}
Jul 24 01:43:55.414496 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:55.414065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:43:55.431000 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:43:55.430946    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d9cdf4547-tglhm" podStartSLOduration=2.020233517 podStartE2EDuration="2.43093036s" totalImagesPullingTime="410.696843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:43:55.428559745 +0000 UTC m=+855.850626581" watchObservedRunningTime="2026-07-24 01:43:55.43093036 +0000 UTC m=+855.852997194"
Jul 24 01:44:06.422504 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:44:06.422473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d9cdf4547-tglhm"
Jul 24 01:44:40.366491 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:44:40.366455    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:44:40.371623 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:44:40.371600    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:44:40.373811 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:44:40.373792    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:44:40.378651 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:44:40.378635    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:45:00.140125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.140089    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"]
Jul 24 01:45:00.147171 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.147139    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"
Jul 24 01:45:00.150426 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.150393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 01:45:00.154877 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.154845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"]
Jul 24 01:45:00.162319 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.162292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"
Jul 24 01:45:00.305333 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.305297    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"]
Jul 24 01:45:00.305480 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:45:00.305372    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317a1758_c518_4615_a147_b46ae78073ce.slice/crio-0cee69fb359c6de9cc441e9f176516f0fd83888a9a2b0cdca38fa1a4eecd4fce WatchSource:0}: Error finding container 0cee69fb359c6de9cc441e9f176516f0fd83888a9a2b0cdca38fa1a4eecd4fce: Status 404 returned error can't find the container with id 0cee69fb359c6de9cc441e9f176516f0fd83888a9a2b0cdca38fa1a4eecd4fce
Jul 24 01:45:00.788653 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:00.788617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq" event={"ID":"317a1758-c518-4615-a147-b46ae78073ce","Type":"ContainerStarted","Data":"0cee69fb359c6de9cc441e9f176516f0fd83888a9a2b0cdca38fa1a4eecd4fce"}
Jul 24 01:45:03.809091 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:03.808984    2564 generic.go:356] "Generic (PLEG): container finished" podID="317a1758-c518-4615-a147-b46ae78073ce" containerID="c505780404cd2d9158fa5fe9ee186f861f6e3be0f32720a15aceaf8ed24075ed" exitCode=0
Jul 24 01:45:03.809462 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:03.809079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq" event={"ID":"317a1758-c518-4615-a147-b46ae78073ce","Type":"ContainerDied","Data":"c505780404cd2d9158fa5fe9ee186f861f6e3be0f32720a15aceaf8ed24075ed"}
Jul 24 01:45:04.955732 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:04.955686    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"
Jul 24 01:45:05.822860 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:05.822828    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq"
Jul 24 01:45:05.823060 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:05.822827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747625-bt8lq" event={"ID":"317a1758-c518-4615-a147-b46ae78073ce","Type":"ContainerDied","Data":"0cee69fb359c6de9cc441e9f176516f0fd83888a9a2b0cdca38fa1a4eecd4fce"}
Jul 24 01:45:05.823060 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:45:05.822961    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cee69fb359c6de9cc441e9f176516f0fd83888a9a2b0cdca38fa1a4eecd4fce"
Jul 24 01:49:40.408735 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:49:40.408616    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:49:40.414836 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:49:40.414813    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:49:40.416178 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:49:40.416152    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:49:40.421587 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:49:40.421566    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:54:40.447754 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:54:40.447722    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:54:40.454312 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:54:40.454290    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:54:40.455223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:54:40.455195    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:54:40.461684 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:54:40.461665    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:56:51.627636 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.627599    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"]
Jul 24 01:56:51.628353 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.628332    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="317a1758-c518-4615-a147-b46ae78073ce" containerName="cleanup"
Jul 24 01:56:51.628436 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.628356    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="317a1758-c518-4615-a147-b46ae78073ce" containerName="cleanup"
Jul 24 01:56:51.628492 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.628483    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="317a1758-c518-4615-a147-b46ae78073ce" containerName="cleanup"
Jul 24 01:56:51.631172 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.631150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.633421 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.633399    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-44fb0de5-serving-cert\""
Jul 24 01:56:51.642963 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.642936    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"]
Jul 24 01:56:51.670984 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.670940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d321462f-48a2-404c-bf8e-7a170992add6-openshift-service-ca\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.670984 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.670989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgdc\" (UniqueName: \"kubernetes.io/projected/d321462f-48a2-404c-bf8e-7a170992add6-kube-api-access-gzgdc\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.671279 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.671187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d321462f-48a2-404c-bf8e-7a170992add6-maas-api-tls\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.771975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.771943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d321462f-48a2-404c-bf8e-7a170992add6-maas-api-tls\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.772197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.771999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d321462f-48a2-404c-bf8e-7a170992add6-openshift-service-ca\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.772197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.772060    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgdc\" (UniqueName: \"kubernetes.io/projected/d321462f-48a2-404c-bf8e-7a170992add6-kube-api-access-gzgdc\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.772646 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.772610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d321462f-48a2-404c-bf8e-7a170992add6-openshift-service-ca\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.774570 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.774545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d321462f-48a2-404c-bf8e-7a170992add6-maas-api-tls\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.780639 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.780613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgdc\" (UniqueName: \"kubernetes.io/projected/d321462f-48a2-404c-bf8e-7a170992add6-kube-api-access-gzgdc\") pod \"maas-api-e2e-ait-44fb0de5-f555585f5-858ds\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:51.944313 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:51.944271    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:52.086400 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.086368    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"]
Jul 24 01:56:52.088109 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:56:52.088082    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd321462f_48a2_404c_bf8e_7a170992add6.slice/crio-7b68a1e2684f40a764117792454cec4ed1bde774b121c4c819ab3b02c4e8dac4 WatchSource:0}: Error finding container 7b68a1e2684f40a764117792454cec4ed1bde774b121c4c819ab3b02c4e8dac4: Status 404 returned error can't find the container with id 7b68a1e2684f40a764117792454cec4ed1bde774b121c4c819ab3b02c4e8dac4
Jul 24 01:56:52.089612 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.089593    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 01:56:52.421071 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.420973    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"]
Jul 24 01:56:52.425901 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.425878    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.428221 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.428183    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-44fb0de5-dockercfg-x8542\""
Jul 24 01:56:52.428221 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.428183    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-44fb0de5\""
Jul 24 01:56:52.437147 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.437111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"]
Jul 24 01:56:52.477806 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.477764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpqd\" (UniqueName: \"kubernetes.io/projected/1b1ae478-40e5-4f35-ac8f-511b8a565052-kube-api-access-grpqd\") pod \"payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.477993 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.477899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1ae478-40e5-4f35-ac8f-511b8a565052-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.578456 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.578422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1ae478-40e5-4f35-ac8f-511b8a565052-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.578658 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.578485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpqd\" (UniqueName: \"kubernetes.io/projected/1b1ae478-40e5-4f35-ac8f-511b8a565052-kube-api-access-grpqd\") pod \"payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.579309 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.579286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1ae478-40e5-4f35-ac8f-511b8a565052-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.588644 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.588614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpqd\" (UniqueName: \"kubernetes.io/projected/1b1ae478-40e5-4f35-ac8f-511b8a565052-kube-api-access-grpqd\") pod \"payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.740471 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.740445    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:52.895339 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.895312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"]
Jul 24 01:56:52.897391 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:56:52.897361    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1ae478_40e5_4f35_ac8f_511b8a565052.slice/crio-170005dd2f13de7006cba910296c03f7f665302bc2382a7a7075037ce54e855f WatchSource:0}: Error finding container 170005dd2f13de7006cba910296c03f7f665302bc2382a7a7075037ce54e855f: Status 404 returned error can't find the container with id 170005dd2f13de7006cba910296c03f7f665302bc2382a7a7075037ce54e855f
Jul 24 01:56:52.930130 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.930079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h" event={"ID":"1b1ae478-40e5-4f35-ac8f-511b8a565052","Type":"ContainerStarted","Data":"170005dd2f13de7006cba910296c03f7f665302bc2382a7a7075037ce54e855f"}
Jul 24 01:56:52.931893 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.931862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds" event={"ID":"d321462f-48a2-404c-bf8e-7a170992add6","Type":"ContainerStarted","Data":"4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170"}
Jul 24 01:56:52.932052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.931916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds" event={"ID":"d321462f-48a2-404c-bf8e-7a170992add6","Type":"ContainerStarted","Data":"7b68a1e2684f40a764117792454cec4ed1bde774b121c4c819ab3b02c4e8dac4"}
Jul 24 01:56:52.932052 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.932046    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:56:52.948106 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:52.948054    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds" podStartSLOduration=1.324589116 podStartE2EDuration="1.947996583s" totalImagesPullingTime="623.407467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:56:52.946933981 +0000 UTC m=+1633.369000855" watchObservedRunningTime="2026-07-24 01:56:52.947996583 +0000 UTC m=+1633.370063418"
Jul 24 01:56:53.940266 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:53.940230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h" event={"ID":"1b1ae478-40e5-4f35-ac8f-511b8a565052","Type":"ContainerStarted","Data":"3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1"}
Jul 24 01:56:53.940665 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:53.940323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:56:53.956040 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:53.955968    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h" podStartSLOduration=1.955953246 podStartE2EDuration="1.955953246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:56:53.955021468 +0000 UTC m=+1634.377088343" watchObservedRunningTime="2026-07-24 01:56:53.955953246 +0000 UTC m=+1634.378020081"
Jul 24 01:56:58.946577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:56:58.946545    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:57:04.949324 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:04.949289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:57:17.162376 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.162338    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"]
Jul 24 01:57:17.165924 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.165906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:17.168092 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.168067    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 01:57:17.173828 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.173799    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"]
Jul 24 01:57:17.310223 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.310189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3102ed88-e106-4092-aeaa-8309272d6bae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x\" (UID: \"3102ed88-e106-4092-aeaa-8309272d6bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:17.410944 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.410905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3102ed88-e106-4092-aeaa-8309272d6bae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x\" (UID: \"3102ed88-e106-4092-aeaa-8309272d6bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:17.411657 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.411637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3102ed88-e106-4092-aeaa-8309272d6bae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x\" (UID: \"3102ed88-e106-4092-aeaa-8309272d6bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:17.480323 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.480291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:17.625173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:17.625136    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"]
Jul 24 01:57:17.627020 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:57:17.626993    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3102ed88_e106_4092_aeaa_8309272d6bae.slice/crio-51004c1d62bea1f86970c787b26a56289fb5f72ff7eb25b09d9fec15b04f80ae WatchSource:0}: Error finding container 51004c1d62bea1f86970c787b26a56289fb5f72ff7eb25b09d9fec15b04f80ae: Status 404 returned error can't find the container with id 51004c1d62bea1f86970c787b26a56289fb5f72ff7eb25b09d9fec15b04f80ae
Jul 24 01:57:18.083961 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:18.083877    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x" event={"ID":"3102ed88-e106-4092-aeaa-8309272d6bae","Type":"ContainerStarted","Data":"9f8b375d634a3f485d14c9708b188531cb09222ce860b9419e28ad7e886fb4fc"}
Jul 24 01:57:18.083961 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:18.083956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x" event={"ID":"3102ed88-e106-4092-aeaa-8309272d6bae","Type":"ContainerStarted","Data":"51004c1d62bea1f86970c787b26a56289fb5f72ff7eb25b09d9fec15b04f80ae"}
Jul 24 01:57:18.098616 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:18.098565    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x" podStartSLOduration=1.098549804 podStartE2EDuration="1.098549804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:57:18.097341803 +0000 UTC m=+1658.519408666" watchObservedRunningTime="2026-07-24 01:57:18.098549804 +0000 UTC m=+1658.520616639"
Jul 24 01:57:19.091143 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:19.091110    2564 generic.go:356] "Generic (PLEG): container finished" podID="3102ed88-e106-4092-aeaa-8309272d6bae" containerID="9f8b375d634a3f485d14c9708b188531cb09222ce860b9419e28ad7e886fb4fc" exitCode=0
Jul 24 01:57:19.091528 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:19.091199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x" event={"ID":"3102ed88-e106-4092-aeaa-8309272d6bae","Type":"ContainerDied","Data":"9f8b375d634a3f485d14c9708b188531cb09222ce860b9419e28ad7e886fb4fc"}
Jul 24 01:57:20.242078 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:20.242051    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:20.338729 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:20.338696    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3102ed88-e106-4092-aeaa-8309272d6bae-maas-api-service-ca\") pod \"3102ed88-e106-4092-aeaa-8309272d6bae\" (UID: \"3102ed88-e106-4092-aeaa-8309272d6bae\") "
Jul 24 01:57:20.339112 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:20.339086    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3102ed88-e106-4092-aeaa-8309272d6bae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3102ed88-e106-4092-aeaa-8309272d6bae" (UID: "3102ed88-e106-4092-aeaa-8309272d6bae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:57:20.440480 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:20.440398    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3102ed88-e106-4092-aeaa-8309272d6bae-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:57:21.105622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:21.105587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x" event={"ID":"3102ed88-e106-4092-aeaa-8309272d6bae","Type":"ContainerDied","Data":"51004c1d62bea1f86970c787b26a56289fb5f72ff7eb25b09d9fec15b04f80ae"}
Jul 24 01:57:21.105622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:21.105606    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"
Jul 24 01:57:21.105836 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:21.105638    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51004c1d62bea1f86970c787b26a56289fb5f72ff7eb25b09d9fec15b04f80ae"
Jul 24 01:57:27.585953 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:27.585916    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"]
Jul 24 01:57:27.586530 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:27.586160    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds" podUID="d321462f-48a2-404c-bf8e-7a170992add6" containerName="maas-api" containerID="cri-o://4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170" gracePeriod=30
Jul 24 01:57:27.644733 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:27.644686    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"]
Jul 24 01:57:27.644954 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:27.644927    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h" podUID="1b1ae478-40e5-4f35-ac8f-511b8a565052" containerName="payload-pre-processing" containerID="cri-o://3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1" gracePeriod=30
Jul 24 01:57:27.908473 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:27.908445    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:57:27.935389 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:27.935351    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:57:28.011138 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.011104    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzgdc\" (UniqueName: \"kubernetes.io/projected/d321462f-48a2-404c-bf8e-7a170992add6-kube-api-access-gzgdc\") pod \"d321462f-48a2-404c-bf8e-7a170992add6\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") "
Jul 24 01:57:28.011138 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.011151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d321462f-48a2-404c-bf8e-7a170992add6-maas-api-tls\") pod \"d321462f-48a2-404c-bf8e-7a170992add6\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") "
Jul 24 01:57:28.011401 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.011183    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d321462f-48a2-404c-bf8e-7a170992add6-openshift-service-ca\") pod \"d321462f-48a2-404c-bf8e-7a170992add6\" (UID: \"d321462f-48a2-404c-bf8e-7a170992add6\") "
Jul 24 01:57:28.011606 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.011583    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d321462f-48a2-404c-bf8e-7a170992add6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d321462f-48a2-404c-bf8e-7a170992add6" (UID: "d321462f-48a2-404c-bf8e-7a170992add6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:57:28.013454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.013384    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d321462f-48a2-404c-bf8e-7a170992add6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d321462f-48a2-404c-bf8e-7a170992add6" (UID: "d321462f-48a2-404c-bf8e-7a170992add6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:57:28.013454 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.013421    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d321462f-48a2-404c-bf8e-7a170992add6-kube-api-access-gzgdc" (OuterVolumeSpecName: "kube-api-access-gzgdc") pod "d321462f-48a2-404c-bf8e-7a170992add6" (UID: "d321462f-48a2-404c-bf8e-7a170992add6"). InnerVolumeSpecName "kube-api-access-gzgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:57:28.112843 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.112743    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grpqd\" (UniqueName: \"kubernetes.io/projected/1b1ae478-40e5-4f35-ac8f-511b8a565052-kube-api-access-grpqd\") pod \"1b1ae478-40e5-4f35-ac8f-511b8a565052\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") "
Jul 24 01:57:28.112843 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.112798    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1ae478-40e5-4f35-ac8f-511b8a565052-plugins-config-volume\") pod \"1b1ae478-40e5-4f35-ac8f-511b8a565052\" (UID: \"1b1ae478-40e5-4f35-ac8f-511b8a565052\") "
Jul 24 01:57:28.113161 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.113105    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d321462f-48a2-404c-bf8e-7a170992add6-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:57:28.113161 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.113123    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgdc\" (UniqueName: \"kubernetes.io/projected/d321462f-48a2-404c-bf8e-7a170992add6-kube-api-access-gzgdc\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:57:28.113161 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.113137    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d321462f-48a2-404c-bf8e-7a170992add6-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:57:28.113293 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.113200    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1ae478-40e5-4f35-ac8f-511b8a565052-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b1ae478-40e5-4f35-ac8f-511b8a565052" (UID: "1b1ae478-40e5-4f35-ac8f-511b8a565052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:57:28.115039 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.115007    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1ae478-40e5-4f35-ac8f-511b8a565052-kube-api-access-grpqd" (OuterVolumeSpecName: "kube-api-access-grpqd") pod "1b1ae478-40e5-4f35-ac8f-511b8a565052" (UID: "1b1ae478-40e5-4f35-ac8f-511b8a565052"). InnerVolumeSpecName "kube-api-access-grpqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:57:28.148746 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.148704    2564 generic.go:356] "Generic (PLEG): container finished" podID="1b1ae478-40e5-4f35-ac8f-511b8a565052" containerID="3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1" exitCode=0
Jul 24 01:57:28.148907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.148753    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"
Jul 24 01:57:28.148907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.148792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h" event={"ID":"1b1ae478-40e5-4f35-ac8f-511b8a565052","Type":"ContainerDied","Data":"3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1"}
Jul 24 01:57:28.148907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.148855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h" event={"ID":"1b1ae478-40e5-4f35-ac8f-511b8a565052","Type":"ContainerDied","Data":"170005dd2f13de7006cba910296c03f7f665302bc2382a7a7075037ce54e855f"}
Jul 24 01:57:28.148907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.148889    2564 scope.go:117] "RemoveContainer" containerID="3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1"
Jul 24 01:57:28.150284 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.150258    2564 generic.go:356] "Generic (PLEG): container finished" podID="d321462f-48a2-404c-bf8e-7a170992add6" containerID="4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170" exitCode=0
Jul 24 01:57:28.150408 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.150281    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds" event={"ID":"d321462f-48a2-404c-bf8e-7a170992add6","Type":"ContainerDied","Data":"4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170"}
Jul 24 01:57:28.150408 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.150307    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"
Jul 24 01:57:28.150408 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.150332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds" event={"ID":"d321462f-48a2-404c-bf8e-7a170992add6","Type":"ContainerDied","Data":"7b68a1e2684f40a764117792454cec4ed1bde774b121c4c819ab3b02c4e8dac4"}
Jul 24 01:57:28.161974 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.161950    2564 scope.go:117] "RemoveContainer" containerID="3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1"
Jul 24 01:57:28.162321 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:57:28.162296    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1\": container with ID starting with 3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1 not found: ID does not exist" containerID="3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1"
Jul 24 01:57:28.162394 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.162334    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1"} err="failed to get container status \"3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1\": rpc error: code = NotFound desc = could not find container \"3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1\": container with ID starting with 3086ff52e18edf522860cdb11f78d78dfe6f357cd07a34b33f2afef261ca63e1 not found: ID does not exist"
Jul 24 01:57:28.162394 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.162358    2564 scope.go:117] "RemoveContainer" containerID="4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170"
Jul 24 01:57:28.178923 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.178900    2564 scope.go:117] "RemoveContainer" containerID="4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170"
Jul 24 01:57:28.179273 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:57:28.179241    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170\": container with ID starting with 4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170 not found: ID does not exist" containerID="4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170"
Jul 24 01:57:28.179418 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.179280    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170"} err="failed to get container status \"4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170\": rpc error: code = NotFound desc = could not find container \"4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170\": container with ID starting with 4db3aedf1c381d5b3f6fdf83bf0422731142571375dc353e1ec636446916a170 not found: ID does not exist"
Jul 24 01:57:28.182156 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.182134    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"]
Jul 24 01:57:28.184553 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.184533    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44fb0de5-f555585f5-858ds"]
Jul 24 01:57:28.193586 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.193559    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"]
Jul 24 01:57:28.197245 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.197225    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-44fb0de5-fc5896ff6-btz5h"]
Jul 24 01:57:28.211375 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.211348    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1ae478-40e5-4f35-ac8f-511b8a565052" path="/var/lib/kubelet/pods/1b1ae478-40e5-4f35-ac8f-511b8a565052/volumes"
Jul 24 01:57:28.211907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.211889    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d321462f-48a2-404c-bf8e-7a170992add6" path="/var/lib/kubelet/pods/d321462f-48a2-404c-bf8e-7a170992add6/volumes"
Jul 24 01:57:28.214514 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.214494    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grpqd\" (UniqueName: \"kubernetes.io/projected/1b1ae478-40e5-4f35-ac8f-511b8a565052-kube-api-access-grpqd\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:57:28.214576 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:28.214518    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1ae478-40e5-4f35-ac8f-511b8a565052-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:57:32.245313 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:32.245272    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"]
Jul 24 01:57:32.247882 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:32.247854    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44fb0de5-wf67x"]
Jul 24 01:57:34.209885 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:34.209854    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3102ed88-e106-4092-aeaa-8309272d6bae" path="/var/lib/kubelet/pods/3102ed88-e106-4092-aeaa-8309272d6bae/volumes"
Jul 24 01:57:36.081011 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.080978    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"]
Jul 24 01:57:36.081478 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081464    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1ae478-40e5-4f35-ac8f-511b8a565052" containerName="payload-pre-processing"
Jul 24 01:57:36.081522 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081480    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1ae478-40e5-4f35-ac8f-511b8a565052" containerName="payload-pre-processing"
Jul 24 01:57:36.081522 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081498    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d321462f-48a2-404c-bf8e-7a170992add6" containerName="maas-api"
Jul 24 01:57:36.081522 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081504    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d321462f-48a2-404c-bf8e-7a170992add6" containerName="maas-api"
Jul 24 01:57:36.081522 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081511    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3102ed88-e106-4092-aeaa-8309272d6bae" containerName="revoke-keys"
Jul 24 01:57:36.081522 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081518    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3102ed88-e106-4092-aeaa-8309272d6bae" containerName="revoke-keys"
Jul 24 01:57:36.081708 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081594    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1ae478-40e5-4f35-ac8f-511b8a565052" containerName="payload-pre-processing"
Jul 24 01:57:36.081708 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081603    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d321462f-48a2-404c-bf8e-7a170992add6" containerName="maas-api"
Jul 24 01:57:36.081708 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.081609    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3102ed88-e106-4092-aeaa-8309272d6bae" containerName="revoke-keys"
Jul 24 01:57:36.086286 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.086265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.088795 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.088777    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3d183c95-serving-cert\""
Jul 24 01:57:36.096555 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.096510    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"]
Jul 24 01:57:36.189768 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.189732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f0880-d123-40c9-8f7e-f6d925a0e792-openshift-service-ca\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.189768 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.189773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bf8\" (UniqueName: \"kubernetes.io/projected/3a8f0880-d123-40c9-8f7e-f6d925a0e792-kube-api-access-t2bf8\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.189982 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.189897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.291201 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.291161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.291424 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.291282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f0880-d123-40c9-8f7e-f6d925a0e792-openshift-service-ca\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.291424 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:57:36.291313    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-serving-cert: secret "maas-api-e2e-migrate-3d183c95-serving-cert" not found
Jul 24 01:57:36.291424 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:57:36.291401    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls podName:3a8f0880-d123-40c9-8f7e-f6d925a0e792 nodeName:}" failed. No retries permitted until 2026-07-24 01:57:36.791379489 +0000 UTC m=+1677.213446303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls") pod "maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" (UID: "3a8f0880-d123-40c9-8f7e-f6d925a0e792") : secret "maas-api-e2e-migrate-3d183c95-serving-cert" not found
Jul 24 01:57:36.291600 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.291447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bf8\" (UniqueName: \"kubernetes.io/projected/3a8f0880-d123-40c9-8f7e-f6d925a0e792-kube-api-access-t2bf8\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.291983 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.291962    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f0880-d123-40c9-8f7e-f6d925a0e792-openshift-service-ca\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.300926 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.300902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bf8\" (UniqueName: \"kubernetes.io/projected/3a8f0880-d123-40c9-8f7e-f6d925a0e792-kube-api-access-t2bf8\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.672069 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.672017    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"]
Jul 24 01:57:36.675946 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.675923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.678670 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.678644    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3d183c95-dockercfg-g45g2\""
Jul 24 01:57:36.678802 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.678764    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3d183c95\""
Jul 24 01:57:36.688809 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.688785    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"]
Jul 24 01:57:36.696477 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.696451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5mng\" (UniqueName: \"kubernetes.io/projected/17100ec7-981e-4140-8bdb-407794a7dec1-kube-api-access-w5mng\") pod \"payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.696598 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.696496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17100ec7-981e-4140-8bdb-407794a7dec1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.705107 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.705080    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"]
Jul 24 01:57:36.708650 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.708633    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.719506 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.719475    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"]
Jul 24 01:57:36.797016 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.796971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.797220 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.797021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5mng\" (UniqueName: \"kubernetes.io/projected/17100ec7-981e-4140-8bdb-407794a7dec1-kube-api-access-w5mng\") pod \"payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.797220 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.797088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f1cba5-0bc2-496c-bcd9-117c0707e487-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") " pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.797220 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.797111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17100ec7-981e-4140-8bdb-407794a7dec1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.797220 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.797179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppl4v\" (UniqueName: \"kubernetes.io/projected/d4f1cba5-0bc2-496c-bcd9-117c0707e487-kube-api-access-ppl4v\") pod \"payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") " pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.797868 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.797845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17100ec7-981e-4140-8bdb-407794a7dec1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.799438 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.799415    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls\") pod \"maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:36.804842 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.804817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5mng\" (UniqueName: \"kubernetes.io/projected/17100ec7-981e-4140-8bdb-407794a7dec1-kube-api-access-w5mng\") pod \"payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.898520 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.898482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f1cba5-0bc2-496c-bcd9-117c0707e487-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") " pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.898520 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.898532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppl4v\" (UniqueName: \"kubernetes.io/projected/d4f1cba5-0bc2-496c-bcd9-117c0707e487-kube-api-access-ppl4v\") pod \"payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") " pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.899253 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.899222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f1cba5-0bc2-496c-bcd9-117c0707e487-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") " pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.905853 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.905829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppl4v\" (UniqueName: \"kubernetes.io/projected/d4f1cba5-0bc2-496c-bcd9-117c0707e487-kube-api-access-ppl4v\") pod \"payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") " pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:36.988719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.988687    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:36.998622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:36.998602    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:37.021803 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:37.021762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:37.161616 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:37.161221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"]
Jul 24 01:57:37.163504 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:57:37.163459    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17100ec7_981e_4140_8bdb_407794a7dec1.slice/crio-d5f4a3c6a02d1626ae9346204c49820a34a206044d029af885c6ef4f14ab7ec9 WatchSource:0}: Error finding container d5f4a3c6a02d1626ae9346204c49820a34a206044d029af885c6ef4f14ab7ec9: Status 404 returned error can't find the container with id d5f4a3c6a02d1626ae9346204c49820a34a206044d029af885c6ef4f14ab7ec9
Jul 24 01:57:37.195745 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:37.195714    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"]
Jul 24 01:57:37.200498 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:57:37.200467    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8f0880_d123_40c9_8f7e_f6d925a0e792.slice/crio-3c3b734fe7f9948ec95081eb861baff357646ab8436e0cb54fdef6d8aef7d2ce WatchSource:0}: Error finding container 3c3b734fe7f9948ec95081eb861baff357646ab8436e0cb54fdef6d8aef7d2ce: Status 404 returned error can't find the container with id 3c3b734fe7f9948ec95081eb861baff357646ab8436e0cb54fdef6d8aef7d2ce
Jul 24 01:57:37.207668 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:37.207635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp" event={"ID":"17100ec7-981e-4140-8bdb-407794a7dec1","Type":"ContainerStarted","Data":"d5f4a3c6a02d1626ae9346204c49820a34a206044d029af885c6ef4f14ab7ec9"}
Jul 24 01:57:37.221594 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:37.221567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"]
Jul 24 01:57:37.222167 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:57:37.222135    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f1cba5_0bc2_496c_bcd9_117c0707e487.slice/crio-a8608243eca5faba7450ca411dc16894b6c2b5e4df21805b23511bb1f8f82f99 WatchSource:0}: Error finding container a8608243eca5faba7450ca411dc16894b6c2b5e4df21805b23511bb1f8f82f99: Status 404 returned error can't find the container with id a8608243eca5faba7450ca411dc16894b6c2b5e4df21805b23511bb1f8f82f99
Jul 24 01:57:38.223624 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.223581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp" event={"ID":"17100ec7-981e-4140-8bdb-407794a7dec1","Type":"ContainerStarted","Data":"9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073"}
Jul 24 01:57:38.224141 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.223770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:38.225392 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.225360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn" event={"ID":"d4f1cba5-0bc2-496c-bcd9-117c0707e487","Type":"ContainerStarted","Data":"8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b"}
Jul 24 01:57:38.225538 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.225412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn" event={"ID":"d4f1cba5-0bc2-496c-bcd9-117c0707e487","Type":"ContainerStarted","Data":"a8608243eca5faba7450ca411dc16894b6c2b5e4df21805b23511bb1f8f82f99"}
Jul 24 01:57:38.225538 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.225438    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:57:38.227467 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.227437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" event={"ID":"3a8f0880-d123-40c9-8f7e-f6d925a0e792","Type":"ContainerStarted","Data":"fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48"}
Jul 24 01:57:38.227582 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.227496    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" event={"ID":"3a8f0880-d123-40c9-8f7e-f6d925a0e792","Type":"ContainerStarted","Data":"3c3b734fe7f9948ec95081eb861baff357646ab8436e0cb54fdef6d8aef7d2ce"}
Jul 24 01:57:38.227582 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.227522    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:38.238489 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.238442    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp" podStartSLOduration=2.238426961 podStartE2EDuration="2.238426961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:57:38.237599659 +0000 UTC m=+1678.659666497" watchObservedRunningTime="2026-07-24 01:57:38.238426961 +0000 UTC m=+1678.660493863"
Jul 24 01:57:38.253830 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.253777    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn" podStartSLOduration=2.253761954 podStartE2EDuration="2.253761954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:57:38.251184966 +0000 UTC m=+1678.673251833" watchObservedRunningTime="2026-07-24 01:57:38.253761954 +0000 UTC m=+1678.675828802"
Jul 24 01:57:38.265964 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:38.265916    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" podStartSLOduration=1.921129001 podStartE2EDuration="2.26590394s" totalImagesPullingTime="344.774939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:57:38.264904748 +0000 UTC m=+1678.686971584" watchObservedRunningTime="2026-07-24 01:57:38.26590394 +0000 UTC m=+1678.687970776"
Jul 24 01:57:44.238537 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:44.238509    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:57:49.235442 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:49.235411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:57:49.235921 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:57:49.235485    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:58:00.944078 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:00.944040    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"]
Jul 24 01:58:00.949020 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:00.949001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:00.952288 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:00.952192    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 01:58:00.956227 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:00.956203    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"]
Jul 24 01:58:01.034727 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:01.034686    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a91e68-a1b0-411c-a304-b70b5fb196ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg\" (UID: \"d0a91e68-a1b0-411c-a304-b70b5fb196ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:01.135817 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:01.135771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a91e68-a1b0-411c-a304-b70b5fb196ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg\" (UID: \"d0a91e68-a1b0-411c-a304-b70b5fb196ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:01.136452 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:01.136429    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a91e68-a1b0-411c-a304-b70b5fb196ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg\" (UID: \"d0a91e68-a1b0-411c-a304-b70b5fb196ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:01.261849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:01.261747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:01.411983 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:01.411952    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"]
Jul 24 01:58:01.413525 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:58:01.413501    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a91e68_a1b0_411c_a304_b70b5fb196ef.slice/crio-9f59829c3e721a485d2e30649332e4e395a43563d3b5105fe857f6d3a4890410 WatchSource:0}: Error finding container 9f59829c3e721a485d2e30649332e4e395a43563d3b5105fe857f6d3a4890410: Status 404 returned error can't find the container with id 9f59829c3e721a485d2e30649332e4e395a43563d3b5105fe857f6d3a4890410
Jul 24 01:58:02.376547 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:02.376414    2564 generic.go:356] "Generic (PLEG): container finished" podID="d0a91e68-a1b0-411c-a304-b70b5fb196ef" containerID="ba3c0dddbfa0214f392777a2e5629e5675eeaa23bf31f25677aa88f24396fa23" exitCode=0
Jul 24 01:58:02.376547 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:02.376498    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg" event={"ID":"d0a91e68-a1b0-411c-a304-b70b5fb196ef","Type":"ContainerDied","Data":"ba3c0dddbfa0214f392777a2e5629e5675eeaa23bf31f25677aa88f24396fa23"}
Jul 24 01:58:02.377079 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:02.376555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg" event={"ID":"d0a91e68-a1b0-411c-a304-b70b5fb196ef","Type":"ContainerStarted","Data":"9f59829c3e721a485d2e30649332e4e395a43563d3b5105fe857f6d3a4890410"}
Jul 24 01:58:03.529539 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:03.529513    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:03.663134 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:03.663013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a91e68-a1b0-411c-a304-b70b5fb196ef-maas-api-service-ca\") pod \"d0a91e68-a1b0-411c-a304-b70b5fb196ef\" (UID: \"d0a91e68-a1b0-411c-a304-b70b5fb196ef\") "
Jul 24 01:58:03.663409 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:03.663382    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a91e68-a1b0-411c-a304-b70b5fb196ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0a91e68-a1b0-411c-a304-b70b5fb196ef" (UID: "d0a91e68-a1b0-411c-a304-b70b5fb196ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:58:03.764679 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:03.764643    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a91e68-a1b0-411c-a304-b70b5fb196ef-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:04.391719 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:04.391688    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"
Jul 24 01:58:04.391925 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:04.391740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg" event={"ID":"d0a91e68-a1b0-411c-a304-b70b5fb196ef","Type":"ContainerDied","Data":"9f59829c3e721a485d2e30649332e4e395a43563d3b5105fe857f6d3a4890410"}
Jul 24 01:58:04.391925 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:04.391799    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f59829c3e721a485d2e30649332e4e395a43563d3b5105fe857f6d3a4890410"
Jul 24 01:58:11.345166 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.345127    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"]
Jul 24 01:58:11.345662 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.345411    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" podUID="3a8f0880-d123-40c9-8f7e-f6d925a0e792" containerName="maas-api" containerID="cri-o://fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48" gracePeriod=30
Jul 24 01:58:11.399096 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.399064    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"]
Jul 24 01:58:11.399353 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.399294    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn" podUID="d4f1cba5-0bc2-496c-bcd9-117c0707e487" containerName="payload-processing" containerID="cri-o://8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b" gracePeriod=30
Jul 24 01:58:11.406868 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.406840    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"]
Jul 24 01:58:11.407092 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.407067    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp" podUID="17100ec7-981e-4140-8bdb-407794a7dec1" containerName="payload-pre-processing" containerID="cri-o://9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073" gracePeriod=30
Jul 24 01:58:11.715199 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.715155    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:58:11.719537 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.719514    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:58:11.745634 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.745606    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppl4v\" (UniqueName: \"kubernetes.io/projected/d4f1cba5-0bc2-496c-bcd9-117c0707e487-kube-api-access-ppl4v\") pod \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") "
Jul 24 01:58:11.745782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.745660    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls\") pod \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") "
Jul 24 01:58:11.745782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.745705    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f0880-d123-40c9-8f7e-f6d925a0e792-openshift-service-ca\") pod \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") "
Jul 24 01:58:11.745782 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.745731    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2bf8\" (UniqueName: \"kubernetes.io/projected/3a8f0880-d123-40c9-8f7e-f6d925a0e792-kube-api-access-t2bf8\") pod \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\" (UID: \"3a8f0880-d123-40c9-8f7e-f6d925a0e792\") "
Jul 24 01:58:11.745946 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.745853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f1cba5-0bc2-496c-bcd9-117c0707e487-plugins-config-volume\") pod \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\" (UID: \"d4f1cba5-0bc2-496c-bcd9-117c0707e487\") "
Jul 24 01:58:11.746382 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.746352    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f1cba5-0bc2-496c-bcd9-117c0707e487-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4f1cba5-0bc2-496c-bcd9-117c0707e487" (UID: "d4f1cba5-0bc2-496c-bcd9-117c0707e487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:58:11.746561 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.746375    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8f0880-d123-40c9-8f7e-f6d925a0e792-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a8f0880-d123-40c9-8f7e-f6d925a0e792" (UID: "3a8f0880-d123-40c9-8f7e-f6d925a0e792"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:58:11.748051 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.747983    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8f0880-d123-40c9-8f7e-f6d925a0e792-kube-api-access-t2bf8" (OuterVolumeSpecName: "kube-api-access-t2bf8") pod "3a8f0880-d123-40c9-8f7e-f6d925a0e792" (UID: "3a8f0880-d123-40c9-8f7e-f6d925a0e792"). InnerVolumeSpecName "kube-api-access-t2bf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:58:11.748371 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.748341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f1cba5-0bc2-496c-bcd9-117c0707e487-kube-api-access-ppl4v" (OuterVolumeSpecName: "kube-api-access-ppl4v") pod "d4f1cba5-0bc2-496c-bcd9-117c0707e487" (UID: "d4f1cba5-0bc2-496c-bcd9-117c0707e487"). InnerVolumeSpecName "kube-api-access-ppl4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:58:11.748510 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.748485    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a8f0880-d123-40c9-8f7e-f6d925a0e792" (UID: "3a8f0880-d123-40c9-8f7e-f6d925a0e792"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:58:11.755717 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.755690    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:58:11.847194 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847155    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5mng\" (UniqueName: \"kubernetes.io/projected/17100ec7-981e-4140-8bdb-407794a7dec1-kube-api-access-w5mng\") pod \"17100ec7-981e-4140-8bdb-407794a7dec1\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") "
Jul 24 01:58:11.847360 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847210    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17100ec7-981e-4140-8bdb-407794a7dec1-plugins-config-volume\") pod \"17100ec7-981e-4140-8bdb-407794a7dec1\" (UID: \"17100ec7-981e-4140-8bdb-407794a7dec1\") "
Jul 24 01:58:11.847564 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847545    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f1cba5-0bc2-496c-bcd9-117c0707e487-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:11.847601 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847571    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppl4v\" (UniqueName: \"kubernetes.io/projected/d4f1cba5-0bc2-496c-bcd9-117c0707e487-kube-api-access-ppl4v\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:11.847601 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847585    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a8f0880-d123-40c9-8f7e-f6d925a0e792-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:11.847677 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847601    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f0880-d123-40c9-8f7e-f6d925a0e792-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:11.847677 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847614    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bf8\" (UniqueName: \"kubernetes.io/projected/3a8f0880-d123-40c9-8f7e-f6d925a0e792-kube-api-access-t2bf8\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:11.847677 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.847654    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17100ec7-981e-4140-8bdb-407794a7dec1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17100ec7-981e-4140-8bdb-407794a7dec1" (UID: "17100ec7-981e-4140-8bdb-407794a7dec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:58:11.849420 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.849403    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17100ec7-981e-4140-8bdb-407794a7dec1-kube-api-access-w5mng" (OuterVolumeSpecName: "kube-api-access-w5mng") pod "17100ec7-981e-4140-8bdb-407794a7dec1" (UID: "17100ec7-981e-4140-8bdb-407794a7dec1"). InnerVolumeSpecName "kube-api-access-w5mng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:58:11.948545 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.948512    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5mng\" (UniqueName: \"kubernetes.io/projected/17100ec7-981e-4140-8bdb-407794a7dec1-kube-api-access-w5mng\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:11.948545 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:11.948547    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17100ec7-981e-4140-8bdb-407794a7dec1-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:12.449817 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.449781    2564 generic.go:356] "Generic (PLEG): container finished" podID="17100ec7-981e-4140-8bdb-407794a7dec1" containerID="9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073" exitCode=0
Jul 24 01:58:12.450300 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.449824    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"
Jul 24 01:58:12.450300 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.449867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp" event={"ID":"17100ec7-981e-4140-8bdb-407794a7dec1","Type":"ContainerDied","Data":"9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073"}
Jul 24 01:58:12.450300 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.449912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp" event={"ID":"17100ec7-981e-4140-8bdb-407794a7dec1","Type":"ContainerDied","Data":"d5f4a3c6a02d1626ae9346204c49820a34a206044d029af885c6ef4f14ab7ec9"}
Jul 24 01:58:12.450300 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.449934    2564 scope.go:117] "RemoveContainer" containerID="9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073"
Jul 24 01:58:12.451326 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.451292    2564 generic.go:356] "Generic (PLEG): container finished" podID="d4f1cba5-0bc2-496c-bcd9-117c0707e487" containerID="8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b" exitCode=0
Jul 24 01:58:12.451391 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.451334    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"
Jul 24 01:58:12.451391 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.451346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn" event={"ID":"d4f1cba5-0bc2-496c-bcd9-117c0707e487","Type":"ContainerDied","Data":"8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b"}
Jul 24 01:58:12.451482 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.451393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn" event={"ID":"d4f1cba5-0bc2-496c-bcd9-117c0707e487","Type":"ContainerDied","Data":"a8608243eca5faba7450ca411dc16894b6c2b5e4df21805b23511bb1f8f82f99"}
Jul 24 01:58:12.452966 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.452946    2564 generic.go:356] "Generic (PLEG): container finished" podID="3a8f0880-d123-40c9-8f7e-f6d925a0e792" containerID="fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48" exitCode=0
Jul 24 01:58:12.453111 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.452998    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"
Jul 24 01:58:12.453111 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.453072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" event={"ID":"3a8f0880-d123-40c9-8f7e-f6d925a0e792","Type":"ContainerDied","Data":"fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48"}
Jul 24 01:58:12.453111 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.453108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc" event={"ID":"3a8f0880-d123-40c9-8f7e-f6d925a0e792","Type":"ContainerDied","Data":"3c3b734fe7f9948ec95081eb861baff357646ab8436e0cb54fdef6d8aef7d2ce"}
Jul 24 01:58:12.460358 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.460340    2564 scope.go:117] "RemoveContainer" containerID="9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073"
Jul 24 01:58:12.460627 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:12.460606    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073\": container with ID starting with 9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073 not found: ID does not exist" containerID="9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073"
Jul 24 01:58:12.460680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.460631    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073"} err="failed to get container status \"9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073\": rpc error: code = NotFound desc = could not find container \"9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073\": container with ID starting with 9a715b9c24e78f0420d6c3e3dd69482ca3e65e148b61e8a9badf4ef5593ab073 not found: ID does not exist"
Jul 24 01:58:12.460680 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.460648    2564 scope.go:117] "RemoveContainer" containerID="8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b"
Jul 24 01:58:12.468849 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.468825    2564 scope.go:117] "RemoveContainer" containerID="8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b"
Jul 24 01:58:12.469255 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:12.469202    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b\": container with ID starting with 8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b not found: ID does not exist" containerID="8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b"
Jul 24 01:58:12.469255 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.469231    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b"} err="failed to get container status \"8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b\": rpc error: code = NotFound desc = could not find container \"8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b\": container with ID starting with 8ee9a507d4f78ff9e108a92d00d47029ed5a582f1985f0efb23d048a9832356b not found: ID does not exist"
Jul 24 01:58:12.469255 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.469252    2564 scope.go:117] "RemoveContainer" containerID="fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48"
Jul 24 01:58:12.472824 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.472802    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"]
Jul 24 01:58:12.477970 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.477948    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3d183c95-6d5856c48-r5lxp"]
Jul 24 01:58:12.478907 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.478892    2564 scope.go:117] "RemoveContainer" containerID="fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48"
Jul 24 01:58:12.479163 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:12.479145    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48\": container with ID starting with fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48 not found: ID does not exist" containerID="fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48"
Jul 24 01:58:12.479230 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.479170    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48"} err="failed to get container status \"fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48\": rpc error: code = NotFound desc = could not find container \"fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48\": container with ID starting with fa7183f2b2c5fcf26933576df29cda4d10ff6d970b6f8aee42c807b2b1907e48 not found: ID does not exist"
Jul 24 01:58:12.487995 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.487970    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"]
Jul 24 01:58:12.491845 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.491826    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3d183c95-8577fb945c-t5drn"]
Jul 24 01:58:12.500750 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.500733    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"]
Jul 24 01:58:12.507514 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:12.507492    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3d183c95-59c68bb5cd-nj8gc"]
Jul 24 01:58:14.210854 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:14.210815    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17100ec7-981e-4140-8bdb-407794a7dec1" path="/var/lib/kubelet/pods/17100ec7-981e-4140-8bdb-407794a7dec1/volumes"
Jul 24 01:58:14.211474 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:14.211453    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a8f0880-d123-40c9-8f7e-f6d925a0e792" path="/var/lib/kubelet/pods/3a8f0880-d123-40c9-8f7e-f6d925a0e792/volumes"
Jul 24 01:58:14.211839 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:14.211820    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f1cba5-0bc2-496c-bcd9-117c0707e487" path="/var/lib/kubelet/pods/d4f1cba5-0bc2-496c-bcd9-117c0707e487/volumes"
Jul 24 01:58:16.018561 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.018515    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"]
Jul 24 01:58:16.021057 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.021034    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3d183c95-8z4lg"]
Jul 24 01:58:16.210661 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.210623    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a91e68-a1b0-411c-a304-b70b5fb196ef" path="/var/lib/kubelet/pods/d0a91e68-a1b0-411c-a304-b70b5fb196ef/volumes"
Jul 24 01:58:16.971728 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.971693    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"]
Jul 24 01:58:16.972153 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972139    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17100ec7-981e-4140-8bdb-407794a7dec1" containerName="payload-pre-processing"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972155    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17100ec7-981e-4140-8bdb-407794a7dec1" containerName="payload-pre-processing"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972166    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f1cba5-0bc2-496c-bcd9-117c0707e487" containerName="payload-processing"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972172    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f1cba5-0bc2-496c-bcd9-117c0707e487" containerName="payload-processing"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972182    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a91e68-a1b0-411c-a304-b70b5fb196ef" containerName="revoke-keys"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972188    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a91e68-a1b0-411c-a304-b70b5fb196ef" containerName="revoke-keys"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972202    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8f0880-d123-40c9-8f7e-f6d925a0e792" containerName="maas-api"
Jul 24 01:58:16.972237 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972210    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8f0880-d123-40c9-8f7e-f6d925a0e792" containerName="maas-api"
Jul 24 01:58:16.972448 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972297    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8f0880-d123-40c9-8f7e-f6d925a0e792" containerName="maas-api"
Jul 24 01:58:16.972448 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972308    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a91e68-a1b0-411c-a304-b70b5fb196ef" containerName="revoke-keys"
Jul 24 01:58:16.972448 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972316    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17100ec7-981e-4140-8bdb-407794a7dec1" containerName="payload-pre-processing"
Jul 24 01:58:16.972448 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.972333    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f1cba5-0bc2-496c-bcd9-117c0707e487" containerName="payload-processing"
Jul 24 01:58:16.976557 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.976537    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.978998 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.978975    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-da5bdfda-openshift-default-dockercfg-brckk\""
Jul 24 01:58:16.988192 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.988164    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"]
Jul 24 01:58:16.995827 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.995802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462a22dd-e0f9-4e71-9628-e3eb949ca179-istiod-ca-cert\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.995975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.995863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-data\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.995975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.995927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-certs\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.995975 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.995956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-podinfo\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.996197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.995978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-socket\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.996197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.996070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-token\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.996197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.996104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-credential-socket\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.996197 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.996123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpmg\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-kube-api-access-kzpmg\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:16.996415 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:16.996200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-envoy\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097401 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-envoy\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462a22dd-e0f9-4e71-9628-e3eb949ca179-istiod-ca-cert\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-data\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-certs\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-podinfo\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-socket\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-token\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-credential-socket\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpmg\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-kube-api-access-kzpmg\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.097889 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-data\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.098335 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.097935    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-certs\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.098335 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.098229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-credential-socket\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.098335 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.098249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-socket\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.098335 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.098289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462a22dd-e0f9-4e71-9628-e3eb949ca179-istiod-ca-cert\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.100508 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.100473    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-envoy\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.100687 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.100667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-podinfo\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.105595 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.105566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-token\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.106008 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.105991    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpmg\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-kube-api-access-kzpmg\") pod \"e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") " pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.289393 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.289290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:17.435566 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.435535    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"]
Jul 24 01:58:17.436750 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:58:17.436726    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462a22dd_e0f9_4e71_9628_e3eb949ca179.slice/crio-557020ab9b2923845a366697fd948d5f3deda981458a016a549e252626596a8f WatchSource:0}: Error finding container 557020ab9b2923845a366697fd948d5f3deda981458a016a549e252626596a8f: Status 404 returned error can't find the container with id 557020ab9b2923845a366697fd948d5f3deda981458a016a549e252626596a8f
Jul 24 01:58:17.439082 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.439045    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:58:17.439175 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.439118    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:58:17.439175 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.439146    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:58:17.491855 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:17.491807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" event={"ID":"462a22dd-e0f9-4e71-9628-e3eb949ca179","Type":"ContainerStarted","Data":"557020ab9b2923845a366697fd948d5f3deda981458a016a549e252626596a8f"}
Jul 24 01:58:18.499438 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:18.499400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" event={"ID":"462a22dd-e0f9-4e71-9628-e3eb949ca179","Type":"ContainerStarted","Data":"843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"}
Jul 24 01:58:18.521341 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:18.521289    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" podStartSLOduration=2.52127299 podStartE2EDuration="2.52127299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:58:18.517937366 +0000 UTC m=+1718.940004213" watchObservedRunningTime="2026-07-24 01:58:18.52127299 +0000 UTC m=+1718.943339882"
Jul 24 01:58:19.290210 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:19.290169    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:19.295365 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:19.295335    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:19.505499 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:19.505472    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:19.506565 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:19.506548    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:20.085776 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.085741    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"]
Jul 24 01:58:20.090306 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.090280    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.092684 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.092664    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-da5bdfda-serving-cert\""
Jul 24 01:58:20.100592 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.100568    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"]
Jul 24 01:58:20.128514 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.128483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.128688 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.128537    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d25d54-e52b-40c6-b053-6c06803c6796-openshift-service-ca\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.128688 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.128584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrv8t\" (UniqueName: \"kubernetes.io/projected/98d25d54-e52b-40c6-b053-6c06803c6796-kube-api-access-vrv8t\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.229519 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.229490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrv8t\" (UniqueName: \"kubernetes.io/projected/98d25d54-e52b-40c6-b053-6c06803c6796-kube-api-access-vrv8t\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.229695 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.229550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.229695 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.229604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d25d54-e52b-40c6-b053-6c06803c6796-openshift-service-ca\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.229779 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:20.229720    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-serving-cert: secret "maas-api-e2e-ait-da5bdfda-serving-cert" not found
Jul 24 01:58:20.229816 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:20.229790    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls podName:98d25d54-e52b-40c6-b053-6c06803c6796 nodeName:}" failed. No retries permitted until 2026-07-24 01:58:20.729770899 +0000 UTC m=+1721.151837718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls") pod "maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" (UID: "98d25d54-e52b-40c6-b053-6c06803c6796") : secret "maas-api-e2e-ait-da5bdfda-serving-cert" not found
Jul 24 01:58:20.230289 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.230269    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d25d54-e52b-40c6-b053-6c06803c6796-openshift-service-ca\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.242311 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.242290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrv8t\" (UniqueName: \"kubernetes.io/projected/98d25d54-e52b-40c6-b053-6c06803c6796-kube-api-access-vrv8t\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.734933 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:20.734890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:20.735534 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:20.735106    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-serving-cert: secret "maas-api-e2e-ait-da5bdfda-serving-cert" not found
Jul 24 01:58:20.735534 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:20.735186    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls podName:98d25d54-e52b-40c6-b053-6c06803c6796 nodeName:}" failed. No retries permitted until 2026-07-24 01:58:21.735164296 +0000 UTC m=+1722.157231125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls") pod "maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" (UID: "98d25d54-e52b-40c6-b053-6c06803c6796") : secret "maas-api-e2e-ait-da5bdfda-serving-cert" not found
Jul 24 01:58:21.745234 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:21.745196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:21.747685 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:21.747656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") pod \"maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:21.903796 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:21.903756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:22.048947 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:22.048914    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"]
Jul 24 01:58:22.050382 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:58:22.050344    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d25d54_e52b_40c6_b053_6c06803c6796.slice/crio-66fd54203ad4145c2b1a2803ff5d1332f3c086ff92895e002a7167d7022e667c WatchSource:0}: Error finding container 66fd54203ad4145c2b1a2803ff5d1332f3c086ff92895e002a7167d7022e667c: Status 404 returned error can't find the container with id 66fd54203ad4145c2b1a2803ff5d1332f3c086ff92895e002a7167d7022e667c
Jul 24 01:58:22.524352 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:22.524309    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" event={"ID":"98d25d54-e52b-40c6-b053-6c06803c6796","Type":"ContainerStarted","Data":"66fd54203ad4145c2b1a2803ff5d1332f3c086ff92895e002a7167d7022e667c"}
Jul 24 01:58:23.531217 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:23.531178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" event={"ID":"98d25d54-e52b-40c6-b053-6c06803c6796","Type":"ContainerStarted","Data":"07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2"}
Jul 24 01:58:23.531608 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:23.531246    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:23.546702 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:23.546547    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" podStartSLOduration=3.063135949 podStartE2EDuration="3.546527176s" totalImagesPullingTime="483.391227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:58:23.546084235 +0000 UTC m=+1723.968151095" watchObservedRunningTime="2026-07-24 01:58:23.546527176 +0000 UTC m=+1723.968594013"
Jul 24 01:58:29.543915 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:29.543888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:33.596673 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:33.596586    2564 generic.go:356] "Generic (PLEG): container finished" podID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerID="843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2" exitCode=0
Jul 24 01:58:33.596673 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:33.596651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" event={"ID":"462a22dd-e0f9-4e71-9628-e3eb949ca179","Type":"ContainerDied","Data":"843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"}
Jul 24 01:58:33.597153 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:33.597020    2564 scope.go:117] "RemoveContainer" containerID="843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"
Jul 24 01:58:33.598086 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:33.598057    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:58:33.598159 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:33.598142    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:58:33.598203 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:33.598193    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 01:58:34.506477 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:34.506448    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:34.604573 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:34.604531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" event={"ID":"462a22dd-e0f9-4e71-9628-e3eb949ca179","Type":"ContainerStarted","Data":"c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774"}
Jul 24 01:58:35.289667 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:35.289630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:35.291044 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:35.291004    2564 patch_prober.go:28] interesting pod/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 24 01:58:35.291185 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:35.291076    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 24 01:58:36.289749 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:36.289707    2564 patch_prober.go:28] interesting pod/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 24 01:58:36.290178 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:36.289773    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 24 01:58:37.290267 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:37.290228    2564 patch_prober.go:28] interesting pod/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 24 01:58:37.290650 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:37.290285    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 24 01:58:38.290505 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:38.290466    2564 patch_prober.go:28] interesting pod/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 24 01:58:38.290921 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:38.290538    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 24 01:58:39.294753 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:39.294721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:39.295143 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:39.295013    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:39.295736 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:39.295719    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:58:56.216756 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.216723    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"]
Jul 24 01:58:56.217363 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.216995    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" podUID="98d25d54-e52b-40c6-b053-6c06803c6796" containerName="maas-api" containerID="cri-o://07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2" gracePeriod=30
Jul 24 01:58:56.485212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.485130    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:56.602305 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.602269    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") pod \"98d25d54-e52b-40c6-b053-6c06803c6796\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") "
Jul 24 01:58:56.602475 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.602405    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrv8t\" (UniqueName: \"kubernetes.io/projected/98d25d54-e52b-40c6-b053-6c06803c6796-kube-api-access-vrv8t\") pod \"98d25d54-e52b-40c6-b053-6c06803c6796\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") "
Jul 24 01:58:56.602475 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.602426    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d25d54-e52b-40c6-b053-6c06803c6796-openshift-service-ca\") pod \"98d25d54-e52b-40c6-b053-6c06803c6796\" (UID: \"98d25d54-e52b-40c6-b053-6c06803c6796\") "
Jul 24 01:58:56.602843 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.602817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d25d54-e52b-40c6-b053-6c06803c6796-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98d25d54-e52b-40c6-b053-6c06803c6796" (UID: "98d25d54-e52b-40c6-b053-6c06803c6796"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:58:56.604436 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.604407    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98d25d54-e52b-40c6-b053-6c06803c6796" (UID: "98d25d54-e52b-40c6-b053-6c06803c6796"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:58:56.604539 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.604499    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d25d54-e52b-40c6-b053-6c06803c6796-kube-api-access-vrv8t" (OuterVolumeSpecName: "kube-api-access-vrv8t") pod "98d25d54-e52b-40c6-b053-6c06803c6796" (UID: "98d25d54-e52b-40c6-b053-6c06803c6796"). InnerVolumeSpecName "kube-api-access-vrv8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:58:56.703125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.703083    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98d25d54-e52b-40c6-b053-6c06803c6796-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:56.703125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.703116    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrv8t\" (UniqueName: \"kubernetes.io/projected/98d25d54-e52b-40c6-b053-6c06803c6796-kube-api-access-vrv8t\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:56.703125 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.703127    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98d25d54-e52b-40c6-b053-6c06803c6796-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:58:56.748893 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.748805    2564 generic.go:356] "Generic (PLEG): container finished" podID="98d25d54-e52b-40c6-b053-6c06803c6796" containerID="07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2" exitCode=0
Jul 24 01:58:56.748893 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.748854    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"
Jul 24 01:58:56.748893 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.748882    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" event={"ID":"98d25d54-e52b-40c6-b053-6c06803c6796","Type":"ContainerDied","Data":"07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2"}
Jul 24 01:58:56.749166 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.748935    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58" event={"ID":"98d25d54-e52b-40c6-b053-6c06803c6796","Type":"ContainerDied","Data":"66fd54203ad4145c2b1a2803ff5d1332f3c086ff92895e002a7167d7022e667c"}
Jul 24 01:58:56.749166 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.748957    2564 scope.go:117] "RemoveContainer" containerID="07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2"
Jul 24 01:58:56.761725 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.761696    2564 scope.go:117] "RemoveContainer" containerID="07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2"
Jul 24 01:58:56.762003 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:58:56.761982    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2\": container with ID starting with 07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2 not found: ID does not exist" containerID="07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2"
Jul 24 01:58:56.762083 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.762014    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2"} err="failed to get container status \"07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2\": rpc error: code = NotFound desc = could not find container \"07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2\": container with ID starting with 07ca8706f33c760aa138ed7227f0e8f9e2893d49f48dd780029977c604e7e8d2 not found: ID does not exist"
Jul 24 01:58:56.775766 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.775735    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"]
Jul 24 01:58:56.779870 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:56.779846    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-da5bdfda-5897bfc45b-m9z58"]
Jul 24 01:58:58.210147 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:58:58.210113    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d25d54-e52b-40c6-b053-6c06803c6796" path="/var/lib/kubelet/pods/98d25d54-e52b-40c6-b053-6c06803c6796/volumes"
Jul 24 01:59:02.143934 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:02.143898    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"]
Jul 24 01:59:02.144440 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:02.144194    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy" containerID="cri-o://c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774" gracePeriod=30
Jul 24 01:59:05.293514 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.293478    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"]
Jul 24 01:59:05.293942 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.293926    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d25d54-e52b-40c6-b053-6c06803c6796" containerName="maas-api"
Jul 24 01:59:05.293989 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.293943    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d25d54-e52b-40c6-b053-6c06803c6796" containerName="maas-api"
Jul 24 01:59:05.294041 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.294014    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d25d54-e52b-40c6-b053-6c06803c6796" containerName="maas-api"
Jul 24 01:59:05.297233 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.297216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.299619 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.299597    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d1346920-serving-cert\""
Jul 24 01:59:05.309277 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.309255    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"]
Jul 24 01:59:05.390339 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.390305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.390548 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.390374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xz8d\" (UniqueName: \"kubernetes.io/projected/56a64e82-a857-4c89-9d4a-67f67567d892-kube-api-access-7xz8d\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.390548 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.390456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a64e82-a857-4c89-9d4a-67f67567d892-openshift-service-ca\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.491407 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.491373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.491655 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.491471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xz8d\" (UniqueName: \"kubernetes.io/projected/56a64e82-a857-4c89-9d4a-67f67567d892-kube-api-access-7xz8d\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.491655 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.491518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a64e82-a857-4c89-9d4a-67f67567d892-openshift-service-ca\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.491655 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:59:05.491524    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-serving-cert: secret "maas-api-e2e-derive-d1346920-serving-cert" not found
Jul 24 01:59:05.491655 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:59:05.491587    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls podName:56a64e82-a857-4c89-9d4a-67f67567d892 nodeName:}" failed. No retries permitted until 2026-07-24 01:59:05.99156976 +0000 UTC m=+1766.413636575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls") pod "maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" (UID: "56a64e82-a857-4c89-9d4a-67f67567d892") : secret "maas-api-e2e-derive-d1346920-serving-cert" not found
Jul 24 01:59:05.492193 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.492168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a64e82-a857-4c89-9d4a-67f67567d892-openshift-service-ca\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.502105 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.502073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xz8d\" (UniqueName: \"kubernetes.io/projected/56a64e82-a857-4c89-9d4a-67f67567d892-kube-api-access-7xz8d\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:05.997817 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:05.997779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:06.000416 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:06.000384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls\") pod \"maas-api-e2e-derive-d1346920-75d6575b49-pp5rb\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:06.209074 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:06.209041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:06.369826 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:06.369797    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"]
Jul 24 01:59:06.371318 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:59:06.371291    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a64e82_a857_4c89_9d4a_67f67567d892.slice/crio-0f59a0da64fba09f512407bad5a042020235548dcdb0e37016111bb055af5782 WatchSource:0}: Error finding container 0f59a0da64fba09f512407bad5a042020235548dcdb0e37016111bb055af5782: Status 404 returned error can't find the container with id 0f59a0da64fba09f512407bad5a042020235548dcdb0e37016111bb055af5782
Jul 24 01:59:06.812280 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:06.812244    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" event={"ID":"56a64e82-a857-4c89-9d4a-67f67567d892","Type":"ContainerStarted","Data":"0f59a0da64fba09f512407bad5a042020235548dcdb0e37016111bb055af5782"}
Jul 24 01:59:07.413458 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.413377    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:59:07.518173 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518138    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-podinfo\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518359 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-socket\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518359 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518205    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462a22dd-e0f9-4e71-9628-e3eb949ca179-istiod-ca-cert\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518359 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518222    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-credential-socket\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518534 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518363    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-token\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518534 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-data\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518534 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518441    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:59:07.518694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518533    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-certs\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:59:07.518694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518579    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzpmg\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-kube-api-access-kzpmg\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-envoy\") pod \"462a22dd-e0f9-4e71-9628-e3eb949ca179\" (UID: \"462a22dd-e0f9-4e71-9628-e3eb949ca179\") "
Jul 24 01:59:07.518694 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:59:07.518950 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518624    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462a22dd-e0f9-4e71-9628-e3eb949ca179-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:59:07.519001 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518946    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-socket\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.519001 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518967    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462a22dd-e0f9-4e71-9628-e3eb949ca179-istiod-ca-cert\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.519001 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.518982    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-credential-socket\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.519182 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.519000    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-workload-certs\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.519399 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.519370    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-data" (OuterVolumeSpecName: "istio-data") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:59:07.520577 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.520546    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-token" (OuterVolumeSpecName: "istio-token") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:59:07.520725 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.520703    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-kube-api-access-kzpmg" (OuterVolumeSpecName: "kube-api-access-kzpmg") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "kube-api-access-kzpmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:59:07.520800 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.520714    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 01:59:07.520855 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.520802    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "462a22dd-e0f9-4e71-9628-e3eb949ca179" (UID: "462a22dd-e0f9-4e71-9628-e3eb949ca179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 01:59:07.619761 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.619716    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-data\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.619761 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.619757    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzpmg\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-kube-api-access-kzpmg\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.619761 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.619770    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-envoy\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.619996 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.619779    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-podinfo\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.619996 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.619787    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462a22dd-e0f9-4e71-9628-e3eb949ca179-istio-token\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:07.820297 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.820262    2564 generic.go:356] "Generic (PLEG): container finished" podID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerID="c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774" exitCode=0
Jul 24 01:59:07.820497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.820314    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"
Jul 24 01:59:07.820497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.820336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" event={"ID":"462a22dd-e0f9-4e71-9628-e3eb949ca179","Type":"ContainerDied","Data":"c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774"}
Jul 24 01:59:07.820497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.820388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z" event={"ID":"462a22dd-e0f9-4e71-9628-e3eb949ca179","Type":"ContainerDied","Data":"557020ab9b2923845a366697fd948d5f3deda981458a016a549e252626596a8f"}
Jul 24 01:59:07.820497 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.820413    2564 scope.go:117] "RemoveContainer" containerID="c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774"
Jul 24 01:59:07.822106 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.822073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" event={"ID":"56a64e82-a857-4c89-9d4a-67f67567d892","Type":"ContainerStarted","Data":"c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0"}
Jul 24 01:59:07.822255 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.822137    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:07.831360 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.831340    2564 scope.go:117] "RemoveContainer" containerID="843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"
Jul 24 01:59:07.840384 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.840358    2564 scope.go:117] "RemoveContainer" containerID="c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774"
Jul 24 01:59:07.840704 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:59:07.840680    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774\": container with ID starting with c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774 not found: ID does not exist" containerID="c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774"
Jul 24 01:59:07.840808 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.840710    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774"} err="failed to get container status \"c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774\": rpc error: code = NotFound desc = could not find container \"c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774\": container with ID starting with c97ea0271f2e49f06bf3345820db265495437e38e259ce924c155cda0a1d8774 not found: ID does not exist"
Jul 24 01:59:07.840808 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.840729    2564 scope.go:117] "RemoveContainer" containerID="843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"
Jul 24 01:59:07.840964 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:59:07.840946    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2\": container with ID starting with 843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2 not found: ID does not exist" containerID="843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"
Jul 24 01:59:07.841020 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.840969    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2"} err="failed to get container status \"843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2\": rpc error: code = NotFound desc = could not find container \"843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2\": container with ID starting with 843ffbf02ff3013788d861bb5a17a9e41dd82c0a2c0ff9274fc151a564ad35c2 not found: ID does not exist"
Jul 24 01:59:07.841500 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.841430    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" podStartSLOduration=2.220095204 podStartE2EDuration="2.841417292s" totalImagesPullingTime="621.322088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:59:07.839097664 +0000 UTC m=+1768.261164525" watchObservedRunningTime="2026-07-24 01:59:07.841417292 +0000 UTC m=+1768.263484128"
Jul 24 01:59:07.853282 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.853254    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"]
Jul 24 01:59:07.857209 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:07.857188    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-da5bdfda-openshift-default-7cc7bc585d-47d6z"]
Jul 24 01:59:08.211250 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:08.211218    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" path="/var/lib/kubelet/pods/462a22dd-e0f9-4e71-9628-e3eb949ca179/volumes"
Jul 24 01:59:13.834943 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:13.834913    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:30.123305 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123262    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"]
Jul 24 01:59:30.123760 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123744    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy"
Jul 24 01:59:30.123760 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123756    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy"
Jul 24 01:59:30.123834 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123785    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy"
Jul 24 01:59:30.123834 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123793    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy"
Jul 24 01:59:30.123896 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123871    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy"
Jul 24 01:59:30.123896 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.123881    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="462a22dd-e0f9-4e71-9628-e3eb949ca179" containerName="istio-proxy"
Jul 24 01:59:30.126155 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.126132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:30.128456 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.128433    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 01:59:30.134311 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.134288    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"]
Jul 24 01:59:30.214208 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.214174    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cba78e2-41dd-4f2d-a1d8-678253cf947c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1346920-jfl6b\" (UID: \"5cba78e2-41dd-4f2d-a1d8-678253cf947c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:30.315571 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.315535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cba78e2-41dd-4f2d-a1d8-678253cf947c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1346920-jfl6b\" (UID: \"5cba78e2-41dd-4f2d-a1d8-678253cf947c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:30.316212 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.316192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cba78e2-41dd-4f2d-a1d8-678253cf947c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d1346920-jfl6b\" (UID: \"5cba78e2-41dd-4f2d-a1d8-678253cf947c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:30.440041 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.439941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:30.612676 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.612631    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"]
Jul 24 01:59:30.614510 ip-10-0-129-127 kubenswrapper[2564]: W0724 01:59:30.614485    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cba78e2_41dd_4f2d_a1d8_678253cf947c.slice/crio-b402a79d05ca203082cbaea59d5ca28a8d37e23501bb21b58f3676c2d542fce6 WatchSource:0}: Error finding container b402a79d05ca203082cbaea59d5ca28a8d37e23501bb21b58f3676c2d542fce6: Status 404 returned error can't find the container with id b402a79d05ca203082cbaea59d5ca28a8d37e23501bb21b58f3676c2d542fce6
Jul 24 01:59:30.961812 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.961776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b" event={"ID":"5cba78e2-41dd-4f2d-a1d8-678253cf947c","Type":"ContainerStarted","Data":"28f4d2328a1dd51c2bda2c7f9565fd4380170ff8e77681e4fac1b2d3b506a877"}
Jul 24 01:59:30.961971 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.961829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b" event={"ID":"5cba78e2-41dd-4f2d-a1d8-678253cf947c","Type":"ContainerStarted","Data":"b402a79d05ca203082cbaea59d5ca28a8d37e23501bb21b58f3676c2d542fce6"}
Jul 24 01:59:30.979462 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:30.979415    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b" podStartSLOduration=0.979399478 podStartE2EDuration="979.399478ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:59:30.977456495 +0000 UTC m=+1791.399523332" watchObservedRunningTime="2026-07-24 01:59:30.979399478 +0000 UTC m=+1791.401466313"
Jul 24 01:59:31.969909 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:31.969878    2564 generic.go:356] "Generic (PLEG): container finished" podID="5cba78e2-41dd-4f2d-a1d8-678253cf947c" containerID="28f4d2328a1dd51c2bda2c7f9565fd4380170ff8e77681e4fac1b2d3b506a877" exitCode=0
Jul 24 01:59:31.970329 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:31.969951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b" event={"ID":"5cba78e2-41dd-4f2d-a1d8-678253cf947c","Type":"ContainerDied","Data":"28f4d2328a1dd51c2bda2c7f9565fd4380170ff8e77681e4fac1b2d3b506a877"}
Jul 24 01:59:33.113359 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.113328    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:33.148486 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.148453    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cba78e2-41dd-4f2d-a1d8-678253cf947c-maas-api-service-ca\") pod \"5cba78e2-41dd-4f2d-a1d8-678253cf947c\" (UID: \"5cba78e2-41dd-4f2d-a1d8-678253cf947c\") "
Jul 24 01:59:33.148829 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.148805    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cba78e2-41dd-4f2d-a1d8-678253cf947c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cba78e2-41dd-4f2d-a1d8-678253cf947c" (UID: "5cba78e2-41dd-4f2d-a1d8-678253cf947c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:59:33.249667 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.249592    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cba78e2-41dd-4f2d-a1d8-678253cf947c-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:33.982622 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.982589    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"
Jul 24 01:59:33.982774 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.982639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b" event={"ID":"5cba78e2-41dd-4f2d-a1d8-678253cf947c","Type":"ContainerDied","Data":"b402a79d05ca203082cbaea59d5ca28a8d37e23501bb21b58f3676c2d542fce6"}
Jul 24 01:59:33.982774 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:33.982685    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b402a79d05ca203082cbaea59d5ca28a8d37e23501bb21b58f3676c2d542fce6"
Jul 24 01:59:40.490773 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.490743    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:59:40.497567 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.497541    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:59:40.500816 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.500795    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 01:59:40.507735 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.507714    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 01:59:40.692728 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.692694    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"]
Jul 24 01:59:40.692981 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.692933    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" podUID="56a64e82-a857-4c89-9d4a-67f67567d892" containerName="maas-api" containerID="cri-o://c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0" gracePeriod=30
Jul 24 01:59:40.952303 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:40.952271    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:41.023968 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.023936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xz8d\" (UniqueName: \"kubernetes.io/projected/56a64e82-a857-4c89-9d4a-67f67567d892-kube-api-access-7xz8d\") pod \"56a64e82-a857-4c89-9d4a-67f67567d892\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") "
Jul 24 01:59:41.024190 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.024083    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls\") pod \"56a64e82-a857-4c89-9d4a-67f67567d892\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") "
Jul 24 01:59:41.024190 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.024129    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a64e82-a857-4c89-9d4a-67f67567d892-openshift-service-ca\") pod \"56a64e82-a857-4c89-9d4a-67f67567d892\" (UID: \"56a64e82-a857-4c89-9d4a-67f67567d892\") "
Jul 24 01:59:41.024483 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.024457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a64e82-a857-4c89-9d4a-67f67567d892-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56a64e82-a857-4c89-9d4a-67f67567d892" (UID: "56a64e82-a857-4c89-9d4a-67f67567d892"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 01:59:41.026360 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.026331    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a64e82-a857-4c89-9d4a-67f67567d892-kube-api-access-7xz8d" (OuterVolumeSpecName: "kube-api-access-7xz8d") pod "56a64e82-a857-4c89-9d4a-67f67567d892" (UID: "56a64e82-a857-4c89-9d4a-67f67567d892"). InnerVolumeSpecName "kube-api-access-7xz8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 01:59:41.026502 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.026411    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56a64e82-a857-4c89-9d4a-67f67567d892" (UID: "56a64e82-a857-4c89-9d4a-67f67567d892"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 01:59:41.027664 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.027639    2564 generic.go:356] "Generic (PLEG): container finished" podID="56a64e82-a857-4c89-9d4a-67f67567d892" containerID="c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0" exitCode=0
Jul 24 01:59:41.027755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.027695    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"
Jul 24 01:59:41.027755 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.027715    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" event={"ID":"56a64e82-a857-4c89-9d4a-67f67567d892","Type":"ContainerDied","Data":"c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0"}
Jul 24 01:59:41.027888 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.027780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb" event={"ID":"56a64e82-a857-4c89-9d4a-67f67567d892","Type":"ContainerDied","Data":"0f59a0da64fba09f512407bad5a042020235548dcdb0e37016111bb055af5782"}
Jul 24 01:59:41.027888 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.027816    2564 scope.go:117] "RemoveContainer" containerID="c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0"
Jul 24 01:59:41.049073 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.049049    2564 scope.go:117] "RemoveContainer" containerID="c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0"
Jul 24 01:59:41.049434 ip-10-0-129-127 kubenswrapper[2564]: E0724 01:59:41.049411    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0\": container with ID starting with c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0 not found: ID does not exist" containerID="c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0"
Jul 24 01:59:41.049513 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.049447    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0"} err="failed to get container status \"c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0\": rpc error: code = NotFound desc = could not find container \"c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0\": container with ID starting with c1d19d0e36adb40ab551dc0670cf580e1cbee444aefe215cc74841e6359c92f0 not found: ID does not exist"
Jul 24 01:59:41.061131 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.061056    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"]
Jul 24 01:59:41.066434 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.066410    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d1346920-75d6575b49-pp5rb"]
Jul 24 01:59:41.125051 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.124996    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xz8d\" (UniqueName: \"kubernetes.io/projected/56a64e82-a857-4c89-9d4a-67f67567d892-kube-api-access-7xz8d\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:41.125226 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.125063    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a64e82-a857-4c89-9d4a-67f67567d892-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:41.125226 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:41.125076    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a64e82-a857-4c89-9d4a-67f67567d892-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 01:59:42.209897 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:42.209867    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a64e82-a857-4c89-9d4a-67f67567d892" path="/var/lib/kubelet/pods/56a64e82-a857-4c89-9d4a-67f67567d892/volumes"
Jul 24 01:59:45.203619 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:45.203579    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"]
Jul 24 01:59:45.206328 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:45.206304    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d1346920-jfl6b"]
Jul 24 01:59:46.210646 ip-10-0-129-127 kubenswrapper[2564]: I0724 01:59:46.210602    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cba78e2-41dd-4f2d-a1d8-678253cf947c" path="/var/lib/kubelet/pods/5cba78e2-41dd-4f2d-a1d8-678253cf947c/volumes"
Jul 24 02:00:00.132743 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.132606    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"]
Jul 24 02:00:00.133824 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.133800    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a64e82-a857-4c89-9d4a-67f67567d892" containerName="maas-api"
Jul 24 02:00:00.133824 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.133826    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a64e82-a857-4c89-9d4a-67f67567d892" containerName="maas-api"
Jul 24 02:00:00.133948 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.133850    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cba78e2-41dd-4f2d-a1d8-678253cf947c" containerName="revoke-keys"
Jul 24 02:00:00.133948 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.133856    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cba78e2-41dd-4f2d-a1d8-678253cf947c" containerName="revoke-keys"
Jul 24 02:00:00.133948 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.133927    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="56a64e82-a857-4c89-9d4a-67f67567d892" containerName="maas-api"
Jul 24 02:00:00.133948 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.133933    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cba78e2-41dd-4f2d-a1d8-678253cf947c" containerName="revoke-keys"
Jul 24 02:00:00.137234 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.137218    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"
Jul 24 02:00:00.139580 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.139548    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:00:00.143905 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.143868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"]
Jul 24 02:00:00.150812 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.150790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"
Jul 24 02:00:00.295445 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:00.295422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"]
Jul 24 02:00:00.296999 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:00:00.296965    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf6eb04_f811_4ebe_847c_d508e6114da0.slice/crio-be8eeff18d14b88d6917431bf39ab6838a5254be1fef7d7ce9853b646f04cac6 WatchSource:0}: Error finding container be8eeff18d14b88d6917431bf39ab6838a5254be1fef7d7ce9853b646f04cac6: Status 404 returned error can't find the container with id be8eeff18d14b88d6917431bf39ab6838a5254be1fef7d7ce9853b646f04cac6
Jul 24 02:00:01.155552 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:01.155518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p" event={"ID":"5bf6eb04-f811-4ebe-847c-d508e6114da0","Type":"ContainerStarted","Data":"f46cce8edab7989e2de09afa5e3b3e6987f82566de8ff2820d10d572f2c26d7e"}
Jul 24 02:00:01.155944 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:01.155576    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p" event={"ID":"5bf6eb04-f811-4ebe-847c-d508e6114da0","Type":"ContainerStarted","Data":"be8eeff18d14b88d6917431bf39ab6838a5254be1fef7d7ce9853b646f04cac6"}
Jul 24 02:00:01.175733 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:01.175663    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p" podStartSLOduration=1.175641827 podStartE2EDuration="1.175641827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:00:01.173354517 +0000 UTC m=+1821.595421391" watchObservedRunningTime="2026-07-24 02:00:01.175641827 +0000 UTC m=+1821.597708664"
Jul 24 02:00:02.162826 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:02.162786    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bf6eb04-f811-4ebe-847c-d508e6114da0" containerID="f46cce8edab7989e2de09afa5e3b3e6987f82566de8ff2820d10d572f2c26d7e" exitCode=0
Jul 24 02:00:02.163263 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:02.162872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p" event={"ID":"5bf6eb04-f811-4ebe-847c-d508e6114da0","Type":"ContainerDied","Data":"f46cce8edab7989e2de09afa5e3b3e6987f82566de8ff2820d10d572f2c26d7e"}
Jul 24 02:00:03.314260 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:03.314224    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"
Jul 24 02:00:04.176707 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:04.176674    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p"
Jul 24 02:00:04.176888 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:04.176717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747640-rfk4p" event={"ID":"5bf6eb04-f811-4ebe-847c-d508e6114da0","Type":"ContainerDied","Data":"be8eeff18d14b88d6917431bf39ab6838a5254be1fef7d7ce9853b646f04cac6"}
Jul 24 02:00:04.176888 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:00:04.176765    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be8eeff18d14b88d6917431bf39ab6838a5254be1fef7d7ce9853b646f04cac6"
Jul 24 02:01:34.792147 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.792067    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"]
Jul 24 02:01:34.792876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.792537    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf6eb04-f811-4ebe-847c-d508e6114da0" containerName="cleanup"
Jul 24 02:01:34.792876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.792551    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf6eb04-f811-4ebe-847c-d508e6114da0" containerName="cleanup"
Jul 24 02:01:34.792876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.792658    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf6eb04-f811-4ebe-847c-d508e6114da0" containerName="cleanup"
Jul 24 02:01:34.795853 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.795830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.799160 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.799137    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-20b5fb51-serving-cert\""
Jul 24 02:01:34.808346 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.808323    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"]
Jul 24 02:01:34.812398 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.812365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b975043-cda0-419e-84e7-fec622ef98e7-openshift-service-ca\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.812547 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.812498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.812618 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.812559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwtm\" (UniqueName: \"kubernetes.io/projected/7b975043-cda0-419e-84e7-fec622ef98e7-kube-api-access-zpwtm\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.913287 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.913244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.913577 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.913548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwtm\" (UniqueName: \"kubernetes.io/projected/7b975043-cda0-419e-84e7-fec622ef98e7-kube-api-access-zpwtm\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.913577 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:01:34.913494    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-serving-cert: secret "maas-api-e2e-mt-20b5fb51-serving-cert" not found
Jul 24 02:01:34.913791 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:01:34.913664    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls podName:7b975043-cda0-419e-84e7-fec622ef98e7 nodeName:}" failed. No retries permitted until 2026-07-24 02:01:35.413639093 +0000 UTC m=+1915.835705923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls") pod "maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" (UID: "7b975043-cda0-419e-84e7-fec622ef98e7") : secret "maas-api-e2e-mt-20b5fb51-serving-cert" not found
Jul 24 02:01:34.913791 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.913692    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b975043-cda0-419e-84e7-fec622ef98e7-openshift-service-ca\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.914596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.914572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b975043-cda0-419e-84e7-fec622ef98e7-openshift-service-ca\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:34.926690 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:34.926664    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwtm\" (UniqueName: \"kubernetes.io/projected/7b975043-cda0-419e-84e7-fec622ef98e7-kube-api-access-zpwtm\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:35.419081 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:35.419010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:35.421517 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:35.421488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls\") pod \"maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:35.707646 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:35.707604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:35.899902 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:35.899858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"]
Jul 24 02:01:35.905463 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:01:35.905435    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b975043_cda0_419e_84e7_fec622ef98e7.slice/crio-d5218e488c67fc62b8fcea28dc004cb4c94076c3e019490ecef5a7c18776f4f8 WatchSource:0}: Error finding container d5218e488c67fc62b8fcea28dc004cb4c94076c3e019490ecef5a7c18776f4f8: Status 404 returned error can't find the container with id d5218e488c67fc62b8fcea28dc004cb4c94076c3e019490ecef5a7c18776f4f8
Jul 24 02:01:36.137797 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.137693    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"]
Jul 24 02:01:36.142868 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.142839    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.145882 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.145851    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-20b5fb51-dockercfg-6rzfs\""
Jul 24 02:01:36.146130 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.146108    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-20b5fb51\""
Jul 24 02:01:36.159083 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.159053    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"]
Jul 24 02:01:36.227412 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.227379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") " pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.227636 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.227426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zfmw\" (UniqueName: \"kubernetes.io/projected/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-kube-api-access-5zfmw\") pod \"payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") " pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.328218 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.328175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") " pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.328218 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.328224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zfmw\" (UniqueName: \"kubernetes.io/projected/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-kube-api-access-5zfmw\") pod \"payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") " pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.329083 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.329059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") " pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.336813 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.336791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zfmw\" (UniqueName: \"kubernetes.io/projected/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-kube-api-access-5zfmw\") pod \"payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") " pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.455374 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.455348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.610988 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.610956    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"]
Jul 24 02:01:36.613297 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:01:36.613252    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d19f5f_6c97_454d_92a9_274c8a89a6e1.slice/crio-db828dde0f382f62338b77f6c50dcf88256d2384da3086104742b05c8a26364a WatchSource:0}: Error finding container db828dde0f382f62338b77f6c50dcf88256d2384da3086104742b05c8a26364a: Status 404 returned error can't find the container with id db828dde0f382f62338b77f6c50dcf88256d2384da3086104742b05c8a26364a
Jul 24 02:01:36.745323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.745274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" event={"ID":"7b975043-cda0-419e-84e7-fec622ef98e7","Type":"ContainerStarted","Data":"1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522"}
Jul 24 02:01:36.745491 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.745346    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:36.745491 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.745370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" event={"ID":"7b975043-cda0-419e-84e7-fec622ef98e7","Type":"ContainerStarted","Data":"d5218e488c67fc62b8fcea28dc004cb4c94076c3e019490ecef5a7c18776f4f8"}
Jul 24 02:01:36.747740 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.747694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2" event={"ID":"e4d19f5f-6c97-454d-92a9-274c8a89a6e1","Type":"ContainerStarted","Data":"fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe"}
Jul 24 02:01:36.747874 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.747786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2" event={"ID":"e4d19f5f-6c97-454d-92a9-274c8a89a6e1","Type":"ContainerStarted","Data":"db828dde0f382f62338b77f6c50dcf88256d2384da3086104742b05c8a26364a"}
Jul 24 02:01:36.747874 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.747844    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:36.765270 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.765220    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" podStartSLOduration=2.308911833 podStartE2EDuration="2.765205912s" totalImagesPullingTime="456.294079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:01:36.761058976 +0000 UTC m=+1917.183125828" watchObservedRunningTime="2026-07-24 02:01:36.765205912 +0000 UTC m=+1917.187272765"
Jul 24 02:01:36.777981 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:36.777926    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2" podStartSLOduration=0.777908738 podStartE2EDuration="777.908738ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:01:36.77650721 +0000 UTC m=+1917.198574047" watchObservedRunningTime="2026-07-24 02:01:36.777908738 +0000 UTC m=+1917.199975589"
Jul 24 02:01:42.762442 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:42.762409    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:47.758998 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:47.758965    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:48.676236 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.676202    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"]
Jul 24 02:01:48.679789 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.679766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:48.682613 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.682597    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:01:48.693506 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.693479    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"]
Jul 24 02:01:48.749348 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.749307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g\" (UID: \"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:48.850500 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.850457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g\" (UID: \"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:48.851222 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.851201    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g\" (UID: \"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:48.990018 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:48.989979    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:49.153357 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:49.153328    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"]
Jul 24 02:01:49.155190 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:01:49.155145    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d87ccc_7f8b_479d_95b3_b7c325ffd3ed.slice/crio-e3c8885b650e78337742c9a87ce622cd970c92ac32ec86c8296dbf7f650b49e1 WatchSource:0}: Error finding container e3c8885b650e78337742c9a87ce622cd970c92ac32ec86c8296dbf7f650b49e1: Status 404 returned error can't find the container with id e3c8885b650e78337742c9a87ce622cd970c92ac32ec86c8296dbf7f650b49e1
Jul 24 02:01:49.833607 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:49.833571    2564 generic.go:356] "Generic (PLEG): container finished" podID="a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed" containerID="be7682a9896cc36c90ca3415dc8767ccec4d5e96aa519747e551586501187807" exitCode=0
Jul 24 02:01:49.833813 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:49.833656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g" event={"ID":"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed","Type":"ContainerDied","Data":"be7682a9896cc36c90ca3415dc8767ccec4d5e96aa519747e551586501187807"}
Jul 24 02:01:49.833813 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:49.833708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g" event={"ID":"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed","Type":"ContainerStarted","Data":"e3c8885b650e78337742c9a87ce622cd970c92ac32ec86c8296dbf7f650b49e1"}
Jul 24 02:01:50.986624 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:50.986594    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:51.069974 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:51.069940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed-maas-api-service-ca\") pod \"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed\" (UID: \"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed\") "
Jul 24 02:01:51.070330 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:51.070307    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed" (UID: "a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:01:51.170953 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:51.170864    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:01:51.848179 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:51.848151    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"
Jul 24 02:01:51.848339 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:51.848154    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g" event={"ID":"a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed","Type":"ContainerDied","Data":"e3c8885b650e78337742c9a87ce622cd970c92ac32ec86c8296dbf7f650b49e1"}
Jul 24 02:01:51.848339 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:51.848272    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c8885b650e78337742c9a87ce622cd970c92ac32ec86c8296dbf7f650b49e1"
Jul 24 02:01:59.371675 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.371632    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"]
Jul 24 02:01:59.372225 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.371886    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" podUID="7b975043-cda0-419e-84e7-fec622ef98e7" containerName="maas-api" containerID="cri-o://1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522" gracePeriod=30
Jul 24 02:01:59.435833 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.435794    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"]
Jul 24 02:01:59.436056 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.436010    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2" podUID="e4d19f5f-6c97-454d-92a9-274c8a89a6e1" containerName="payload-processing" containerID="cri-o://fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe" gracePeriod=30
Jul 24 02:01:59.695462 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.695433    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:59.723462 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.723438    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:59.850883 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.850840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls\") pod \"7b975043-cda0-419e-84e7-fec622ef98e7\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") "
Jul 24 02:01:59.851105 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.850908    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpwtm\" (UniqueName: \"kubernetes.io/projected/7b975043-cda0-419e-84e7-fec622ef98e7-kube-api-access-zpwtm\") pod \"7b975043-cda0-419e-84e7-fec622ef98e7\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") "
Jul 24 02:01:59.851105 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.850925    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-plugins-config-volume\") pod \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") "
Jul 24 02:01:59.851105 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.851068    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b975043-cda0-419e-84e7-fec622ef98e7-openshift-service-ca\") pod \"7b975043-cda0-419e-84e7-fec622ef98e7\" (UID: \"7b975043-cda0-419e-84e7-fec622ef98e7\") "
Jul 24 02:01:59.851301 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.851114    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zfmw\" (UniqueName: \"kubernetes.io/projected/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-kube-api-access-5zfmw\") pod \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\" (UID: \"e4d19f5f-6c97-454d-92a9-274c8a89a6e1\") "
Jul 24 02:01:59.851371 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.851344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4d19f5f-6c97-454d-92a9-274c8a89a6e1" (UID: "e4d19f5f-6c97-454d-92a9-274c8a89a6e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:01:59.851464 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.851433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b975043-cda0-419e-84e7-fec622ef98e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b975043-cda0-419e-84e7-fec622ef98e7" (UID: "7b975043-cda0-419e-84e7-fec622ef98e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:01:59.851545 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.851523    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:01:59.851545 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.851545    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b975043-cda0-419e-84e7-fec622ef98e7-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:01:59.853214 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.853189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b975043-cda0-419e-84e7-fec622ef98e7" (UID: "7b975043-cda0-419e-84e7-fec622ef98e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:01:59.853317 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.853237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b975043-cda0-419e-84e7-fec622ef98e7-kube-api-access-zpwtm" (OuterVolumeSpecName: "kube-api-access-zpwtm") pod "7b975043-cda0-419e-84e7-fec622ef98e7" (UID: "7b975043-cda0-419e-84e7-fec622ef98e7"). InnerVolumeSpecName "kube-api-access-zpwtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:01:59.853317 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.853272    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-kube-api-access-5zfmw" (OuterVolumeSpecName: "kube-api-access-5zfmw") pod "e4d19f5f-6c97-454d-92a9-274c8a89a6e1" (UID: "e4d19f5f-6c97-454d-92a9-274c8a89a6e1"). InnerVolumeSpecName "kube-api-access-5zfmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:01:59.895672 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.895640    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b975043-cda0-419e-84e7-fec622ef98e7" containerID="1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522" exitCode=0
Jul 24 02:01:59.895831 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.895688    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"
Jul 24 02:01:59.895831 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.895725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" event={"ID":"7b975043-cda0-419e-84e7-fec622ef98e7","Type":"ContainerDied","Data":"1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522"}
Jul 24 02:01:59.895831 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.895776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw" event={"ID":"7b975043-cda0-419e-84e7-fec622ef98e7","Type":"ContainerDied","Data":"d5218e488c67fc62b8fcea28dc004cb4c94076c3e019490ecef5a7c18776f4f8"}
Jul 24 02:01:59.895831 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.895806    2564 scope.go:117] "RemoveContainer" containerID="1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522"
Jul 24 02:01:59.897165 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.897142    2564 generic.go:356] "Generic (PLEG): container finished" podID="e4d19f5f-6c97-454d-92a9-274c8a89a6e1" containerID="fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe" exitCode=0
Jul 24 02:01:59.897283 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.897180    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"
Jul 24 02:01:59.897283 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.897227    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2" event={"ID":"e4d19f5f-6c97-454d-92a9-274c8a89a6e1","Type":"ContainerDied","Data":"fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe"}
Jul 24 02:01:59.897405 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.897275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2" event={"ID":"e4d19f5f-6c97-454d-92a9-274c8a89a6e1","Type":"ContainerDied","Data":"db828dde0f382f62338b77f6c50dcf88256d2384da3086104742b05c8a26364a"}
Jul 24 02:01:59.905816 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.905789    2564 scope.go:117] "RemoveContainer" containerID="1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522"
Jul 24 02:01:59.906193 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:01:59.906149    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522\": container with ID starting with 1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522 not found: ID does not exist" containerID="1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522"
Jul 24 02:01:59.906281 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.906193    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522"} err="failed to get container status \"1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522\": rpc error: code = NotFound desc = could not find container \"1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522\": container with ID starting with 1d0ef2e762e71ce9be25be5719c196333231e45b6e8980e1a45bc000fc2fc522 not found: ID does not exist"
Jul 24 02:01:59.906281 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.906220    2564 scope.go:117] "RemoveContainer" containerID="fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe"
Jul 24 02:01:59.915337 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.915313    2564 scope.go:117] "RemoveContainer" containerID="fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe"
Jul 24 02:01:59.915651 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:01:59.915633    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe\": container with ID starting with fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe not found: ID does not exist" containerID="fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe"
Jul 24 02:01:59.915721 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.915664    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe"} err="failed to get container status \"fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe\": rpc error: code = NotFound desc = could not find container \"fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe\": container with ID starting with fbac7c1cab609dfeadfdcc2d707448232b89d4b19bf9f961ffb4e5f7035634fe not found: ID does not exist"
Jul 24 02:01:59.936492 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.936460    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"]
Jul 24 02:01:59.947823 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.947786    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-20b5fb51-7659d8c55c-964nw"]
Jul 24 02:01:59.952169 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.952146    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b975043-cda0-419e-84e7-fec622ef98e7-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:01:59.952169 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.952169    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpwtm\" (UniqueName: \"kubernetes.io/projected/7b975043-cda0-419e-84e7-fec622ef98e7-kube-api-access-zpwtm\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:01:59.952336 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.952179    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zfmw\" (UniqueName: \"kubernetes.io/projected/e4d19f5f-6c97-454d-92a9-274c8a89a6e1-kube-api-access-5zfmw\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:01:59.963952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.963923    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"]
Jul 24 02:01:59.977259 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:01:59.977233    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20b5fb51-69db4fb66b-z8rm2"]
Jul 24 02:02:00.210859 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:00.210818    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b975043-cda0-419e-84e7-fec622ef98e7" path="/var/lib/kubelet/pods/7b975043-cda0-419e-84e7-fec622ef98e7/volumes"
Jul 24 02:02:00.211248 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:00.211226    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d19f5f-6c97-454d-92a9-274c8a89a6e1" path="/var/lib/kubelet/pods/e4d19f5f-6c97-454d-92a9-274c8a89a6e1/volumes"
Jul 24 02:02:03.851545 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:03.851505    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"]
Jul 24 02:02:03.857870 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:03.857845    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20b5fb51-h6d8g"]
Jul 24 02:02:04.210477 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:04.210445    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed" path="/var/lib/kubelet/pods/a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed/volumes"
Jul 24 02:02:37.302900 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.302816    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"]
Jul 24 02:02:37.303391 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303275    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b975043-cda0-419e-84e7-fec622ef98e7" containerName="maas-api"
Jul 24 02:02:37.303391 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303286    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b975043-cda0-419e-84e7-fec622ef98e7" containerName="maas-api"
Jul 24 02:02:37.303391 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303295    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d19f5f-6c97-454d-92a9-274c8a89a6e1" containerName="payload-processing"
Jul 24 02:02:37.303391 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303301    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d19f5f-6c97-454d-92a9-274c8a89a6e1" containerName="payload-processing"
Jul 24 02:02:37.303391 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303307    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed" containerName="revoke-keys"
Jul 24 02:02:37.303391 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303314    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed" containerName="revoke-keys"
Jul 24 02:02:37.303584 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303399    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d19f5f-6c97-454d-92a9-274c8a89a6e1" containerName="payload-processing"
Jul 24 02:02:37.303584 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303406    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d87ccc-7f8b-479d-95b3-b7c325ffd3ed" containerName="revoke-keys"
Jul 24 02:02:37.303584 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.303417    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b975043-cda0-419e-84e7-fec622ef98e7" containerName="maas-api"
Jul 24 02:02:37.306597 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.306579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.309563 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.309544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d043f7ca-serving-cert\""
Jul 24 02:02:37.326756 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.326725    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"]
Jul 24 02:02:37.406918 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.406880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb907a-a4ea-4452-a8b5-56837d3908d7-openshift-service-ca\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.407140 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.406959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.407140 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.406997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsgf\" (UniqueName: \"kubernetes.io/projected/6ebb907a-a4ea-4452-a8b5-56837d3908d7-kube-api-access-9qsgf\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.508062 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.507993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb907a-a4ea-4452-a8b5-56837d3908d7-openshift-service-ca\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.508237 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.508097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.508237 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.508120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsgf\" (UniqueName: \"kubernetes.io/projected/6ebb907a-a4ea-4452-a8b5-56837d3908d7-kube-api-access-9qsgf\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.508311 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:02:37.508257    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-serving-cert: secret "maas-api-e2e-mt-d043f7ca-serving-cert" not found
Jul 24 02:02:37.508367 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:02:37.508353    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls podName:6ebb907a-a4ea-4452-a8b5-56837d3908d7 nodeName:}" failed. No retries permitted until 2026-07-24 02:02:38.008330582 +0000 UTC m=+1978.430397410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls") pod "maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" (UID: "6ebb907a-a4ea-4452-a8b5-56837d3908d7") : secret "maas-api-e2e-mt-d043f7ca-serving-cert" not found
Jul 24 02:02:37.508763 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.508744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb907a-a4ea-4452-a8b5-56837d3908d7-openshift-service-ca\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:37.530673 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:37.530645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsgf\" (UniqueName: \"kubernetes.io/projected/6ebb907a-a4ea-4452-a8b5-56837d3908d7-kube-api-access-9qsgf\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:38.012819 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:38.012769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:38.015413 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:38.015384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls\") pod \"maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:38.216814 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:38.216777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:38.397576 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:38.397551    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"]
Jul 24 02:02:38.399363 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:02:38.399333    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebb907a_a4ea_4452_a8b5_56837d3908d7.slice/crio-33648c2b55fe2bc9d7ea4d0b5e8c5083f93ebbaf4b0933827962d4316452b11b WatchSource:0}: Error finding container 33648c2b55fe2bc9d7ea4d0b5e8c5083f93ebbaf4b0933827962d4316452b11b: Status 404 returned error can't find the container with id 33648c2b55fe2bc9d7ea4d0b5e8c5083f93ebbaf4b0933827962d4316452b11b
Jul 24 02:02:38.400850 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:38.400832    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 02:02:39.136845 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:39.136812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" event={"ID":"6ebb907a-a4ea-4452-a8b5-56837d3908d7","Type":"ContainerStarted","Data":"883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd"}
Jul 24 02:02:39.137045 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:39.136871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" event={"ID":"6ebb907a-a4ea-4452-a8b5-56837d3908d7","Type":"ContainerStarted","Data":"33648c2b55fe2bc9d7ea4d0b5e8c5083f93ebbaf4b0933827962d4316452b11b"}
Jul 24 02:02:39.137045 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:39.137014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:02:39.165539 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:39.165473    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" podStartSLOduration=1.642628424 podStartE2EDuration="2.165449941s" totalImagesPullingTime="522.821517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:02:39.163085883 +0000 UTC m=+1979.585152743" watchObservedRunningTime="2026-07-24 02:02:39.165449941 +0000 UTC m=+1979.587516779"
Jul 24 02:02:45.148738 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:02:45.148705    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:03:16.405821 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.405786    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"]
Jul 24 02:03:16.409305 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.409286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:16.411882 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.411853    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:03:16.419676 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.419604    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"]
Jul 24 02:03:16.477995 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.477959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24\" (UID: \"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:16.579320 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.579281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24\" (UID: \"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:16.579925 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.579905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24\" (UID: \"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:16.723517 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.723479    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:16.868560 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:16.868526    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"]
Jul 24 02:03:16.869896 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:03:16.869865    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa1a22f_52e2_4f25_9bbe_d34c9f29292d.slice/crio-863e369141632f431e87ef6f29b69e37381b7bc7dfd3f95d8dc1a36c0b429a83 WatchSource:0}: Error finding container 863e369141632f431e87ef6f29b69e37381b7bc7dfd3f95d8dc1a36c0b429a83: Status 404 returned error can't find the container with id 863e369141632f431e87ef6f29b69e37381b7bc7dfd3f95d8dc1a36c0b429a83
Jul 24 02:03:17.360963 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:17.360875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24" event={"ID":"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d","Type":"ContainerStarted","Data":"7474280fdd19f39597b26185c5193b452347eef6780b40e9697f85908e274c9c"}
Jul 24 02:03:17.360963 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:17.360930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24" event={"ID":"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d","Type":"ContainerStarted","Data":"863e369141632f431e87ef6f29b69e37381b7bc7dfd3f95d8dc1a36c0b429a83"}
Jul 24 02:03:17.385574 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:17.385524    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24" podStartSLOduration=1.385506088 podStartE2EDuration="1.385506088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:03:17.384555568 +0000 UTC m=+2017.806622404" watchObservedRunningTime="2026-07-24 02:03:17.385506088 +0000 UTC m=+2017.807572925"
Jul 24 02:03:18.368497 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:18.368461    2564 generic.go:356] "Generic (PLEG): container finished" podID="3fa1a22f-52e2-4f25-9bbe-d34c9f29292d" containerID="7474280fdd19f39597b26185c5193b452347eef6780b40e9697f85908e274c9c" exitCode=0
Jul 24 02:03:18.368972 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:18.368546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24" event={"ID":"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d","Type":"ContainerDied","Data":"7474280fdd19f39597b26185c5193b452347eef6780b40e9697f85908e274c9c"}
Jul 24 02:03:19.518829 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:19.518808    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:19.607384 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:19.607347    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d-maas-api-service-ca\") pod \"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d\" (UID: \"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d\") "
Jul 24 02:03:19.607720 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:19.607698    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fa1a22f-52e2-4f25-9bbe-d34c9f29292d" (UID: "3fa1a22f-52e2-4f25-9bbe-d34c9f29292d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:03:19.708886 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:19.708851    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:03:20.382388 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:20.382358    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"
Jul 24 02:03:20.382388 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:20.382368    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24" event={"ID":"3fa1a22f-52e2-4f25-9bbe-d34c9f29292d","Type":"ContainerDied","Data":"863e369141632f431e87ef6f29b69e37381b7bc7dfd3f95d8dc1a36c0b429a83"}
Jul 24 02:03:20.382603 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:20.382419    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863e369141632f431e87ef6f29b69e37381b7bc7dfd3f95d8dc1a36c0b429a83"
Jul 24 02:03:31.884612 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:31.884576    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"]
Jul 24 02:03:31.885132 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:31.884779    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" podUID="6ebb907a-a4ea-4452-a8b5-56837d3908d7" containerName="maas-api" containerID="cri-o://883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd" gracePeriod=30
Jul 24 02:03:32.147393 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.147315    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:03:32.337358 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.337324    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb907a-a4ea-4452-a8b5-56837d3908d7-openshift-service-ca\") pod \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") "
Jul 24 02:03:32.337560 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.337374    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls\") pod \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") "
Jul 24 02:03:32.337560 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.337513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qsgf\" (UniqueName: \"kubernetes.io/projected/6ebb907a-a4ea-4452-a8b5-56837d3908d7-kube-api-access-9qsgf\") pod \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\" (UID: \"6ebb907a-a4ea-4452-a8b5-56837d3908d7\") "
Jul 24 02:03:32.337731 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.337704    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebb907a-a4ea-4452-a8b5-56837d3908d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ebb907a-a4ea-4452-a8b5-56837d3908d7" (UID: "6ebb907a-a4ea-4452-a8b5-56837d3908d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:03:32.337913 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.337893    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb907a-a4ea-4452-a8b5-56837d3908d7-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:03:32.339731 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.339706    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebb907a-a4ea-4452-a8b5-56837d3908d7-kube-api-access-9qsgf" (OuterVolumeSpecName: "kube-api-access-9qsgf") pod "6ebb907a-a4ea-4452-a8b5-56837d3908d7" (UID: "6ebb907a-a4ea-4452-a8b5-56837d3908d7"). InnerVolumeSpecName "kube-api-access-9qsgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:03:32.339828 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.339727    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ebb907a-a4ea-4452-a8b5-56837d3908d7" (UID: "6ebb907a-a4ea-4452-a8b5-56837d3908d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:03:32.439241 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.439157    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebb907a-a4ea-4452-a8b5-56837d3908d7-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:03:32.439241 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.439186    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qsgf\" (UniqueName: \"kubernetes.io/projected/6ebb907a-a4ea-4452-a8b5-56837d3908d7-kube-api-access-9qsgf\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:03:32.463332 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.463298    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ebb907a-a4ea-4452-a8b5-56837d3908d7" containerID="883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd" exitCode=0
Jul 24 02:03:32.463510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.463372    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"
Jul 24 02:03:32.463510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.463391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" event={"ID":"6ebb907a-a4ea-4452-a8b5-56837d3908d7","Type":"ContainerDied","Data":"883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd"}
Jul 24 02:03:32.463510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.463467    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86" event={"ID":"6ebb907a-a4ea-4452-a8b5-56837d3908d7","Type":"ContainerDied","Data":"33648c2b55fe2bc9d7ea4d0b5e8c5083f93ebbaf4b0933827962d4316452b11b"}
Jul 24 02:03:32.463510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.463491    2564 scope.go:117] "RemoveContainer" containerID="883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd"
Jul 24 02:03:32.473932 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.473911    2564 scope.go:117] "RemoveContainer" containerID="883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd"
Jul 24 02:03:32.474224 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:03:32.474204    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd\": container with ID starting with 883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd not found: ID does not exist" containerID="883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd"
Jul 24 02:03:32.474310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.474235    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd"} err="failed to get container status \"883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd\": rpc error: code = NotFound desc = could not find container \"883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd\": container with ID starting with 883965fbe28f807aea36ab5a6a200460cc6005dc50fc345732fb398623f5d8fd not found: ID does not exist"
Jul 24 02:03:32.486252 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.486221    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"]
Jul 24 02:03:32.488811 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:32.488790    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d043f7ca-6796b9fc74-hcb86"]
Jul 24 02:03:34.209898 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:34.209860    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebb907a-a4ea-4452-a8b5-56837d3908d7" path="/var/lib/kubelet/pods/6ebb907a-a4ea-4452-a8b5-56837d3908d7/volumes"
Jul 24 02:03:36.487803 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:36.487773    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"]
Jul 24 02:03:36.490273 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:36.490254    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d043f7ca-fmh24"]
Jul 24 02:03:38.209941 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:38.209907    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa1a22f-52e2-4f25-9bbe-d34c9f29292d" path="/var/lib/kubelet/pods/3fa1a22f-52e2-4f25-9bbe-d34c9f29292d/volumes"
Jul 24 02:03:42.124172 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:03:42.124138    2564 scope.go:117] "RemoveContainer" containerID="9f8b375d634a3f485d14c9708b188531cb09222ce860b9419e28ad7e886fb4fc"
Jul 24 02:04:40.532580 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:04:40.532486    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:04:40.546982 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:04:40.546949    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:04:40.552279 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:04:40.552257    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:04:40.558768 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:04:40.558747    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:04:42.153754 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:04:42.153713    2564 scope.go:117] "RemoveContainer" containerID="ba3c0dddbfa0214f392777a2e5629e5675eeaa23bf31f25677aa88f24396fa23"
Jul 24 02:05:26.992320 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.992282    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"]
Jul 24 02:05:26.992944 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.992926    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebb907a-a4ea-4452-a8b5-56837d3908d7" containerName="maas-api"
Jul 24 02:05:26.992996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.992948    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebb907a-a4ea-4452-a8b5-56837d3908d7" containerName="maas-api"
Jul 24 02:05:26.992996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.992959    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa1a22f-52e2-4f25-9bbe-d34c9f29292d" containerName="revoke-keys"
Jul 24 02:05:26.992996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.992969    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa1a22f-52e2-4f25-9bbe-d34c9f29292d" containerName="revoke-keys"
Jul 24 02:05:26.993114 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.993105    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebb907a-a4ea-4452-a8b5-56837d3908d7" containerName="maas-api"
Jul 24 02:05:26.993157 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.993120    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa1a22f-52e2-4f25-9bbe-d34c9f29292d" containerName="revoke-keys"
Jul 24 02:05:26.996601 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.996579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:26.999162 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:26.999132    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b9895f-serving-cert\""
Jul 24 02:05:27.007370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.007348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"]
Jul 24 02:05:27.101808 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.101760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7cl\" (UniqueName: \"kubernetes.io/projected/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-kube-api-access-gh7cl\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.102008 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.101825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.102008 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.101873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-maas-api-tls\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.202765 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.202719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7cl\" (UniqueName: \"kubernetes.io/projected/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-kube-api-access-gh7cl\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.202985 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.202781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.202985 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.202900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-maas-api-tls\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.203403 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.203383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.205426 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.205402    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-maas-api-tls\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.220957 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.220922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7cl\" (UniqueName: \"kubernetes.io/projected/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-kube-api-access-gh7cl\") pod \"maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.309952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.309849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:27.457889 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:27.457859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"]
Jul 24 02:05:27.459255 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:05:27.459229    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71ccd34_6c44_455e_8c4c_0b0140ff9da1.slice/crio-eebc3f204f61c8f556217665a2beae9f9108a22a0ee627abfcb8c87696efab07 WatchSource:0}: Error finding container eebc3f204f61c8f556217665a2beae9f9108a22a0ee627abfcb8c87696efab07: Status 404 returned error can't find the container with id eebc3f204f61c8f556217665a2beae9f9108a22a0ee627abfcb8c87696efab07
Jul 24 02:05:28.176326 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:28.176236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk" event={"ID":"f71ccd34-6c44-455e-8c4c-0b0140ff9da1","Type":"ContainerStarted","Data":"3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9"}
Jul 24 02:05:28.176326 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:28.176298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:28.176326 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:28.176324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk" event={"ID":"f71ccd34-6c44-455e-8c4c-0b0140ff9da1","Type":"ContainerStarted","Data":"eebc3f204f61c8f556217665a2beae9f9108a22a0ee627abfcb8c87696efab07"}
Jul 24 02:05:28.195899 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:28.195845    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk" podStartSLOduration=1.760891661 podStartE2EDuration="2.195829437s" totalImagesPullingTime="434.937776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:05:28.194170248 +0000 UTC m=+2148.616237111" watchObservedRunningTime="2026-07-24 02:05:28.195829437 +0000 UTC m=+2148.617896273"
Jul 24 02:05:34.188938 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.188857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:05:34.888894 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.888862    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"]
Jul 24 02:05:34.892863 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.892838    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:34.895625 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.895602    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2df67f-serving-cert\""
Jul 24 02:05:34.904345 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.904318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"]
Jul 24 02:05:34.985792 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.985760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq4c\" (UniqueName: \"kubernetes.io/projected/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-kube-api-access-bwq4c\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:34.985792 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.985804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:34.986116 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:34.985836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.087481 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.087444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq4c\" (UniqueName: \"kubernetes.io/projected/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-kube-api-access-bwq4c\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.087686 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.087490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.087686 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.087520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.087686 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:05:35.087598    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-serving-cert: secret "maas-api-e2e-inf-b-2df67f-serving-cert" not found
Jul 24 02:05:35.087686 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:05:35.087680    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls podName:ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c nodeName:}" failed. No retries permitted until 2026-07-24 02:05:35.587663616 +0000 UTC m=+2156.009730431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls") pod "maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" (UID: "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c") : secret "maas-api-e2e-inf-b-2df67f-serving-cert" not found
Jul 24 02:05:35.088190 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.088173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.097239 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.097209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq4c\" (UniqueName: \"kubernetes.io/projected/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-kube-api-access-bwq4c\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.593588 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.593551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.596107 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.596081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.805697 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.805646    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:35.959310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:35.959267    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"]
Jul 24 02:05:35.960748 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:05:35.960717    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0f9b4e_b02f_4eb4_ae61_ef6a89fbf82c.slice/crio-9c6f7e02ca83964518a0bec7443c71e540bed8e16283842a44161ce6e8438caf WatchSource:0}: Error finding container 9c6f7e02ca83964518a0bec7443c71e540bed8e16283842a44161ce6e8438caf: Status 404 returned error can't find the container with id 9c6f7e02ca83964518a0bec7443c71e540bed8e16283842a44161ce6e8438caf
Jul 24 02:05:36.236150 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:36.236113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" event={"ID":"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c","Type":"ContainerStarted","Data":"9c6f7e02ca83964518a0bec7443c71e540bed8e16283842a44161ce6e8438caf"}
Jul 24 02:05:37.243246 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:37.243208    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" event={"ID":"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c","Type":"ContainerStarted","Data":"e0216c577f5e42a61a48471bd773c6b9d0f03f76a9d594e5e5af0e83fad7fe7f"}
Jul 24 02:05:37.243671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:37.243351    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:05:37.261308 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:37.261245    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" podStartSLOduration=2.583167447 podStartE2EDuration="3.261227545s" totalImagesPullingTime="678.060098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:05:37.259332286 +0000 UTC m=+2157.681399142" watchObservedRunningTime="2026-07-24 02:05:37.261227545 +0000 UTC m=+2157.683294380"
Jul 24 02:05:42.176042 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:42.175982    2564 scope.go:117] "RemoveContainer" containerID="28f4d2328a1dd51c2bda2c7f9565fd4380170ff8e77681e4fac1b2d3b506a877"
Jul 24 02:05:43.254358 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:05:43.254329    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:06:49.987270 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:49.987236    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"]
Jul 24 02:06:49.991216 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:49.991190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:49.994139 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:49.994119    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:06:49.999642 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:49.999559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"]
Jul 24 02:06:50.091501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:50.091471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eae40dc-70d5-4ece-acc2-d66542cb13a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b9895f-94m64\" (UID: \"2eae40dc-70d5-4ece-acc2-d66542cb13a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:50.192243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:50.192209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eae40dc-70d5-4ece-acc2-d66542cb13a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b9895f-94m64\" (UID: \"2eae40dc-70d5-4ece-acc2-d66542cb13a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:50.192855 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:50.192836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eae40dc-70d5-4ece-acc2-d66542cb13a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b9895f-94m64\" (UID: \"2eae40dc-70d5-4ece-acc2-d66542cb13a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:50.303493 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:50.303411    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:50.448049 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:50.448003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"]
Jul 24 02:06:50.450279 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:06:50.450244    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eae40dc_70d5_4ece_acc2_d66542cb13a3.slice/crio-96e56b60da5dda20575506668f1bcdd7331eb0236eb69f1b6a0ca3b56a90fa90 WatchSource:0}: Error finding container 96e56b60da5dda20575506668f1bcdd7331eb0236eb69f1b6a0ca3b56a90fa90: Status 404 returned error can't find the container with id 96e56b60da5dda20575506668f1bcdd7331eb0236eb69f1b6a0ca3b56a90fa90
Jul 24 02:06:50.684809 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:50.684767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64" event={"ID":"2eae40dc-70d5-4ece-acc2-d66542cb13a3","Type":"ContainerStarted","Data":"96e56b60da5dda20575506668f1bcdd7331eb0236eb69f1b6a0ca3b56a90fa90"}
Jul 24 02:06:51.692783 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:51.692691    2564 generic.go:356] "Generic (PLEG): container finished" podID="2eae40dc-70d5-4ece-acc2-d66542cb13a3" containerID="31ec673f441abce75a7fa47c190306165c7157f65024b51fec566199805f84f8" exitCode=0
Jul 24 02:06:51.692783 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:51.692769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64" event={"ID":"2eae40dc-70d5-4ece-acc2-d66542cb13a3","Type":"ContainerDied","Data":"31ec673f441abce75a7fa47c190306165c7157f65024b51fec566199805f84f8"}
Jul 24 02:06:52.844155 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:52.844129    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:52.922136 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:52.922097    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eae40dc-70d5-4ece-acc2-d66542cb13a3-maas-api-service-ca\") pod \"2eae40dc-70d5-4ece-acc2-d66542cb13a3\" (UID: \"2eae40dc-70d5-4ece-acc2-d66542cb13a3\") "
Jul 24 02:06:52.922766 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:52.922727    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eae40dc-70d5-4ece-acc2-d66542cb13a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eae40dc-70d5-4ece-acc2-d66542cb13a3" (UID: "2eae40dc-70d5-4ece-acc2-d66542cb13a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:06:52.922968 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:52.922939    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eae40dc-70d5-4ece-acc2-d66542cb13a3-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:06:53.706017 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:53.705981    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"
Jul 24 02:06:53.706017 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:53.706002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64" event={"ID":"2eae40dc-70d5-4ece-acc2-d66542cb13a3","Type":"ContainerDied","Data":"96e56b60da5dda20575506668f1bcdd7331eb0236eb69f1b6a0ca3b56a90fa90"}
Jul 24 02:06:53.706255 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:06:53.706083    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96e56b60da5dda20575506668f1bcdd7331eb0236eb69f1b6a0ca3b56a90fa90"
Jul 24 02:07:00.453309 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.453274    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"]
Jul 24 02:07:00.453791 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.453482    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk" podUID="f71ccd34-6c44-455e-8c4c-0b0140ff9da1" containerName="maas-api" containerID="cri-o://3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9" gracePeriod=30
Jul 24 02:07:00.719881 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.719811    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:07:00.751868 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.751833    2564 generic.go:356] "Generic (PLEG): container finished" podID="f71ccd34-6c44-455e-8c4c-0b0140ff9da1" containerID="3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9" exitCode=0
Jul 24 02:07:00.752049 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.751878    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"
Jul 24 02:07:00.752049 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.751892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk" event={"ID":"f71ccd34-6c44-455e-8c4c-0b0140ff9da1","Type":"ContainerDied","Data":"3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9"}
Jul 24 02:07:00.752049 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.751947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk" event={"ID":"f71ccd34-6c44-455e-8c4c-0b0140ff9da1","Type":"ContainerDied","Data":"eebc3f204f61c8f556217665a2beae9f9108a22a0ee627abfcb8c87696efab07"}
Jul 24 02:07:00.752049 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.751977    2564 scope.go:117] "RemoveContainer" containerID="3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9"
Jul 24 02:07:00.761632 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.761613    2564 scope.go:117] "RemoveContainer" containerID="3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9"
Jul 24 02:07:00.761880 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:07:00.761860    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9\": container with ID starting with 3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9 not found: ID does not exist" containerID="3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9"
Jul 24 02:07:00.761962 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.761887    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9"} err="failed to get container status \"3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9\": rpc error: code = NotFound desc = could not find container \"3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9\": container with ID starting with 3122e14a053ded079f75568860b6041843a9cc1fdc4aacfeb003adc2a9bfecc9 not found: ID does not exist"
Jul 24 02:07:00.798843 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.798812    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-maas-api-tls\") pod \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") "
Jul 24 02:07:00.798982 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.798859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh7cl\" (UniqueName: \"kubernetes.io/projected/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-kube-api-access-gh7cl\") pod \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") "
Jul 24 02:07:00.799039 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.798993    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-openshift-service-ca\") pod \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\" (UID: \"f71ccd34-6c44-455e-8c4c-0b0140ff9da1\") "
Jul 24 02:07:00.799372 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.799341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f71ccd34-6c44-455e-8c4c-0b0140ff9da1" (UID: "f71ccd34-6c44-455e-8c4c-0b0140ff9da1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:07:00.800980 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.800952    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-kube-api-access-gh7cl" (OuterVolumeSpecName: "kube-api-access-gh7cl") pod "f71ccd34-6c44-455e-8c4c-0b0140ff9da1" (UID: "f71ccd34-6c44-455e-8c4c-0b0140ff9da1"). InnerVolumeSpecName "kube-api-access-gh7cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:07:00.801086 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.801005    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f71ccd34-6c44-455e-8c4c-0b0140ff9da1" (UID: "f71ccd34-6c44-455e-8c4c-0b0140ff9da1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:07:00.900666 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.900634    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:00.900666 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.900664    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7cl\" (UniqueName: \"kubernetes.io/projected/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-kube-api-access-gh7cl\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:00.900666 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:00.900676    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71ccd34-6c44-455e-8c4c-0b0140ff9da1-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:01.076591 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:01.076556    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"]
Jul 24 02:07:01.080793 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:01.080763    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b9895f-7f9f94fb45-qdfpk"]
Jul 24 02:07:02.210295 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:02.210259    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71ccd34-6c44-455e-8c4c-0b0140ff9da1" path="/var/lib/kubelet/pods/f71ccd34-6c44-455e-8c4c-0b0140ff9da1/volumes"
Jul 24 02:07:05.073601 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:05.073561    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"]
Jul 24 02:07:05.078286 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:05.078260    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b9895f-94m64"]
Jul 24 02:07:06.210789 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:06.210749    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eae40dc-70d5-4ece-acc2-d66542cb13a3" path="/var/lib/kubelet/pods/2eae40dc-70d5-4ece-acc2-d66542cb13a3/volumes"
Jul 24 02:07:12.031936 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.031899    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"]
Jul 24 02:07:12.032412 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.032398    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71ccd34-6c44-455e-8c4c-0b0140ff9da1" containerName="maas-api"
Jul 24 02:07:12.032412 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.032413    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71ccd34-6c44-455e-8c4c-0b0140ff9da1" containerName="maas-api"
Jul 24 02:07:12.032487 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.032432    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eae40dc-70d5-4ece-acc2-d66542cb13a3" containerName="revoke-keys"
Jul 24 02:07:12.032487 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.032439    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eae40dc-70d5-4ece-acc2-d66542cb13a3" containerName="revoke-keys"
Jul 24 02:07:12.032553 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.032506    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71ccd34-6c44-455e-8c4c-0b0140ff9da1" containerName="maas-api"
Jul 24 02:07:12.032553 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.032517    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eae40dc-70d5-4ece-acc2-d66542cb13a3" containerName="revoke-keys"
Jul 24 02:07:12.037289 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.037268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:12.039984 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.039963    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:07:12.052521 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.052489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"]
Jul 24 02:07:12.208661 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.208626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08554700-eb26-4d35-9b1a-39380dbada10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9\" (UID: \"08554700-eb26-4d35-9b1a-39380dbada10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:12.309642 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.309524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08554700-eb26-4d35-9b1a-39380dbada10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9\" (UID: \"08554700-eb26-4d35-9b1a-39380dbada10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:12.310289 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.310264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08554700-eb26-4d35-9b1a-39380dbada10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9\" (UID: \"08554700-eb26-4d35-9b1a-39380dbada10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:12.347670 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.347635    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:12.502457 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.502425    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"]
Jul 24 02:07:12.504171 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:07:12.504140    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08554700_eb26_4d35_9b1a_39380dbada10.slice/crio-2b2bf56dff7e2262fc3c9149b4016182d775b6da710d10aa5848d1b1bd164827 WatchSource:0}: Error finding container 2b2bf56dff7e2262fc3c9149b4016182d775b6da710d10aa5848d1b1bd164827: Status 404 returned error can't find the container with id 2b2bf56dff7e2262fc3c9149b4016182d775b6da710d10aa5848d1b1bd164827
Jul 24 02:07:12.829992 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:12.829951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9" event={"ID":"08554700-eb26-4d35-9b1a-39380dbada10","Type":"ContainerStarted","Data":"2b2bf56dff7e2262fc3c9149b4016182d775b6da710d10aa5848d1b1bd164827"}
Jul 24 02:07:13.837991 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:13.837949    2564 generic.go:356] "Generic (PLEG): container finished" podID="08554700-eb26-4d35-9b1a-39380dbada10" containerID="4197618a4f18635b350d956712611f6c34d53a555f0d87b870b914711b371f99" exitCode=0
Jul 24 02:07:13.838441 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:13.838077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9" event={"ID":"08554700-eb26-4d35-9b1a-39380dbada10","Type":"ContainerDied","Data":"4197618a4f18635b350d956712611f6c34d53a555f0d87b870b914711b371f99"}
Jul 24 02:07:14.987996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:14.987970    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:15.035262 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:15.035230    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08554700-eb26-4d35-9b1a-39380dbada10-maas-api-service-ca\") pod \"08554700-eb26-4d35-9b1a-39380dbada10\" (UID: \"08554700-eb26-4d35-9b1a-39380dbada10\") "
Jul 24 02:07:15.035634 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:15.035613    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08554700-eb26-4d35-9b1a-39380dbada10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08554700-eb26-4d35-9b1a-39380dbada10" (UID: "08554700-eb26-4d35-9b1a-39380dbada10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:07:15.136662 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:15.136571    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08554700-eb26-4d35-9b1a-39380dbada10-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:15.858220 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:15.858189    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"
Jul 24 02:07:15.858424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:15.858231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9" event={"ID":"08554700-eb26-4d35-9b1a-39380dbada10","Type":"ContainerDied","Data":"2b2bf56dff7e2262fc3c9149b4016182d775b6da710d10aa5848d1b1bd164827"}
Jul 24 02:07:15.858424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:15.858300    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b2bf56dff7e2262fc3c9149b4016182d775b6da710d10aa5848d1b1bd164827"
Jul 24 02:07:22.639297 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.639254    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"]
Jul 24 02:07:22.639774 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.639533    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" podUID="ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" containerName="maas-api" containerID="cri-o://e0216c577f5e42a61a48471bd773c6b9d0f03f76a9d594e5e5af0e83fad7fe7f" gracePeriod=30
Jul 24 02:07:22.905047 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.904994    2564 generic.go:356] "Generic (PLEG): container finished" podID="ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" containerID="e0216c577f5e42a61a48471bd773c6b9d0f03f76a9d594e5e5af0e83fad7fe7f" exitCode=0
Jul 24 02:07:22.905216 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.905076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" event={"ID":"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c","Type":"ContainerDied","Data":"e0216c577f5e42a61a48471bd773c6b9d0f03f76a9d594e5e5af0e83fad7fe7f"}
Jul 24 02:07:22.905216 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.905141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc" event={"ID":"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c","Type":"ContainerDied","Data":"9c6f7e02ca83964518a0bec7443c71e540bed8e16283842a44161ce6e8438caf"}
Jul 24 02:07:22.905216 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.905165    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c6f7e02ca83964518a0bec7443c71e540bed8e16283842a44161ce6e8438caf"
Jul 24 02:07:22.910586 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:22.910563    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:07:23.002356 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.002323    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwq4c\" (UniqueName: \"kubernetes.io/projected/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-kube-api-access-bwq4c\") pod \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") "
Jul 24 02:07:23.002526 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.002430    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-openshift-service-ca\") pod \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") "
Jul 24 02:07:23.002526 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.002483    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls\") pod \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\" (UID: \"ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c\") "
Jul 24 02:07:23.002833 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.002808    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" (UID: "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:07:23.004522 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.004497    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" (UID: "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:07:23.004625 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.004522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-kube-api-access-bwq4c" (OuterVolumeSpecName: "kube-api-access-bwq4c") pod "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" (UID: "ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c"). InnerVolumeSpecName "kube-api-access-bwq4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:07:23.103385 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.103347    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:23.103385 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.103379    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwq4c\" (UniqueName: \"kubernetes.io/projected/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-kube-api-access-bwq4c\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:23.103385 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.103389    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:07:23.911061 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.911013    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"
Jul 24 02:07:23.933377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.933344    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"]
Jul 24 02:07:23.937158 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:23.937130    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2df67f-6dc85f4bb4-wnmhc"]
Jul 24 02:07:24.210480 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:24.210445    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" path="/var/lib/kubelet/pods/ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c/volumes"
Jul 24 02:07:27.102483 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:27.102441    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"]
Jul 24 02:07:27.105308 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:27.105283    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2df67f-p9rd9"]
Jul 24 02:07:28.210710 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:28.210673    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08554700-eb26-4d35-9b1a-39380dbada10" path="/var/lib/kubelet/pods/08554700-eb26-4d35-9b1a-39380dbada10/volumes"
Jul 24 02:07:37.576665 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.576631    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"]
Jul 24 02:07:37.577367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.577102    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08554700-eb26-4d35-9b1a-39380dbada10" containerName="revoke-keys"
Jul 24 02:07:37.577367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.577115    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08554700-eb26-4d35-9b1a-39380dbada10" containerName="revoke-keys"
Jul 24 02:07:37.577367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.577142    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" containerName="maas-api"
Jul 24 02:07:37.577367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.577148    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" containerName="maas-api"
Jul 24 02:07:37.577367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.577217    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08554700-eb26-4d35-9b1a-39380dbada10" containerName="revoke-keys"
Jul 24 02:07:37.577367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.577227    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0f9b4e-b02f-4eb4-ae61-ef6a89fbf82c" containerName="maas-api"
Jul 24 02:07:37.581535 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.581510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.584529 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.584506    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-834f2e-serving-cert\""
Jul 24 02:07:37.596424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.596393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"]
Jul 24 02:07:37.648700 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.648657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.648888 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.648776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-openshift-service-ca\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.648888 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.648809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrqr\" (UniqueName: \"kubernetes.io/projected/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-kube-api-access-znrqr\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.749786 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.749726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.749995 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.749878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-openshift-service-ca\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.749995 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.749913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrqr\" (UniqueName: \"kubernetes.io/projected/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-kube-api-access-znrqr\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.749995 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:07:37.749914    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-serving-cert: secret "maas-api-e2e-api-a-834f2e-serving-cert" not found
Jul 24 02:07:37.750218 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:07:37.750001    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls podName:f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc nodeName:}" failed. No retries permitted until 2026-07-24 02:07:38.249979291 +0000 UTC m=+2278.672046126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls") pod "maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" (UID: "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc") : secret "maas-api-e2e-api-a-834f2e-serving-cert" not found
Jul 24 02:07:37.750652 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.750631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-openshift-service-ca\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:37.762077 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:37.762017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrqr\" (UniqueName: \"kubernetes.io/projected/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-kube-api-access-znrqr\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:38.255534 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.255495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:38.258596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.258558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls\") pod \"maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:38.260509 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.260478    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"]
Jul 24 02:07:38.264378 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.264356    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.267119 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.267092    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-834f2e\""
Jul 24 02:07:38.267245 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.267097    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-834f2e-dockercfg-sb222\""
Jul 24 02:07:38.274757 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.274728    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"]
Jul 24 02:07:38.279856 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.279825    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"]
Jul 24 02:07:38.283765 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.283730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.305042 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.304995    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"]
Jul 24 02:07:38.356959 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.356916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7d9932-6f44-46ca-9a09-726588af7961-plugins-config-volume\") pod \"payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") " pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.357194 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.356967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxn6\" (UniqueName: \"kubernetes.io/projected/3233e4e9-b8e7-4c6e-a44d-11fde439549a-kube-api-access-fkxn6\") pod \"payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.357194 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.357103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3233e4e9-b8e7-4c6e-a44d-11fde439549a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.357194 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.357157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfg2\" (UniqueName: \"kubernetes.io/projected/0c7d9932-6f44-46ca-9a09-726588af7961-kube-api-access-kcfg2\") pod \"payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") " pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.458594 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.458547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3233e4e9-b8e7-4c6e-a44d-11fde439549a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.458594 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.458601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfg2\" (UniqueName: \"kubernetes.io/projected/0c7d9932-6f44-46ca-9a09-726588af7961-kube-api-access-kcfg2\") pod \"payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") " pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.458843 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.458656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7d9932-6f44-46ca-9a09-726588af7961-plugins-config-volume\") pod \"payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") " pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.458843 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.458690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxn6\" (UniqueName: \"kubernetes.io/projected/3233e4e9-b8e7-4c6e-a44d-11fde439549a-kube-api-access-fkxn6\") pod \"payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.459418 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.459396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3233e4e9-b8e7-4c6e-a44d-11fde439549a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.459584 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.459566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7d9932-6f44-46ca-9a09-726588af7961-plugins-config-volume\") pod \"payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") " pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.467692 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.467644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxn6\" (UniqueName: \"kubernetes.io/projected/3233e4e9-b8e7-4c6e-a44d-11fde439549a-kube-api-access-fkxn6\") pod \"payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.467933 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.467784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfg2\" (UniqueName: \"kubernetes.io/projected/0c7d9932-6f44-46ca-9a09-726588af7961-kube-api-access-kcfg2\") pod \"payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") " pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.491542 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.491510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:38.579487 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.579453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:38.595427 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.595385    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:38.658174 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.658126    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"]
Jul 24 02:07:38.661769 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:07:38.661699    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f44ff8_2d57_4ba3_b413_2a3f0bcc76bc.slice/crio-976825aae7a3c3842916286d2cae3fd2d4be0c07b5c2a770a88f7cab5b2bbb37 WatchSource:0}: Error finding container 976825aae7a3c3842916286d2cae3fd2d4be0c07b5c2a770a88f7cab5b2bbb37: Status 404 returned error can't find the container with id 976825aae7a3c3842916286d2cae3fd2d4be0c07b5c2a770a88f7cab5b2bbb37
Jul 24 02:07:38.666972 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.666642    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 02:07:38.760411 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:07:38.760373    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3233e4e9_b8e7_4c6e_a44d_11fde439549a.slice/crio-7c72fe8e2af8fba9acf5c9d23e215dadbbd8fae1c400a65ee507a813eddebd5a WatchSource:0}: Error finding container 7c72fe8e2af8fba9acf5c9d23e215dadbbd8fae1c400a65ee507a813eddebd5a: Status 404 returned error can't find the container with id 7c72fe8e2af8fba9acf5c9d23e215dadbbd8fae1c400a65ee507a813eddebd5a
Jul 24 02:07:38.762691 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.762462    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"]
Jul 24 02:07:38.787604 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:38.787568    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"]
Jul 24 02:07:38.789330 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:07:38.789294    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7d9932_6f44_46ca_9a09_726588af7961.slice/crio-89f4c62064ed44c583b7e4303a0f78ab96b2552b07323c61795a3ee638aa7fdc WatchSource:0}: Error finding container 89f4c62064ed44c583b7e4303a0f78ab96b2552b07323c61795a3ee638aa7fdc: Status 404 returned error can't find the container with id 89f4c62064ed44c583b7e4303a0f78ab96b2552b07323c61795a3ee638aa7fdc
Jul 24 02:07:39.010489 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.010446    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk" event={"ID":"3233e4e9-b8e7-4c6e-a44d-11fde439549a","Type":"ContainerStarted","Data":"87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3"}
Jul 24 02:07:39.010686 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.010515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk" event={"ID":"3233e4e9-b8e7-4c6e-a44d-11fde439549a","Type":"ContainerStarted","Data":"7c72fe8e2af8fba9acf5c9d23e215dadbbd8fae1c400a65ee507a813eddebd5a"}
Jul 24 02:07:39.010686 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.010617    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:39.012792 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.012757    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t" event={"ID":"0c7d9932-6f44-46ca-9a09-726588af7961","Type":"ContainerStarted","Data":"167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce"}
Jul 24 02:07:39.012963 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.012817    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t" event={"ID":"0c7d9932-6f44-46ca-9a09-726588af7961","Type":"ContainerStarted","Data":"89f4c62064ed44c583b7e4303a0f78ab96b2552b07323c61795a3ee638aa7fdc"}
Jul 24 02:07:39.013210 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.013174    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:39.015339 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.015311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" event={"ID":"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc","Type":"ContainerStarted","Data":"976825aae7a3c3842916286d2cae3fd2d4be0c07b5c2a770a88f7cab5b2bbb37"}
Jul 24 02:07:39.028047 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.027969    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk" podStartSLOduration=1.02795066 podStartE2EDuration="1.02795066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:07:39.026776046 +0000 UTC m=+2279.448842909" watchObservedRunningTime="2026-07-24 02:07:39.02795066 +0000 UTC m=+2279.450017495"
Jul 24 02:07:39.046157 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:39.046101    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t" podStartSLOduration=1.04608163 podStartE2EDuration="1.04608163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:07:39.043994446 +0000 UTC m=+2279.466061310" watchObservedRunningTime="2026-07-24 02:07:39.04608163 +0000 UTC m=+2279.468148466"
Jul 24 02:07:40.023873 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:40.023837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" event={"ID":"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc","Type":"ContainerStarted","Data":"34407157a3c689200462a84eca8c7724f9ce44212de2e1cf248011b05560357a"}
Jul 24 02:07:40.024352 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:40.023900    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:40.042430 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:40.042371    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" podStartSLOduration=2.521818455 podStartE2EDuration="3.042355226s" totalImagesPullingTime="520.536771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:07:40.040291966 +0000 UTC m=+2280.462358825" watchObservedRunningTime="2026-07-24 02:07:40.042355226 +0000 UTC m=+2280.464422061"
Jul 24 02:07:45.705627 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.705591    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"]
Jul 24 02:07:45.712659 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.712627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.715854 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.715829    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a955c2-serving-cert\""
Jul 24 02:07:45.724785 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.724753    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"]
Jul 24 02:07:45.827481 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.827439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dbc4e3-7aff-4101-b492-83dde20786ef-openshift-service-ca\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.827642 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.827569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.827642 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.827625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm2c\" (UniqueName: \"kubernetes.io/projected/a3dbc4e3-7aff-4101-b492-83dde20786ef-kube-api-access-lcm2c\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.928735 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.928690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.928956 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.928762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm2c\" (UniqueName: \"kubernetes.io/projected/a3dbc4e3-7aff-4101-b492-83dde20786ef-kube-api-access-lcm2c\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.928956 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.928809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dbc4e3-7aff-4101-b492-83dde20786ef-openshift-service-ca\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.928956 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:07:45.928903    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-serving-cert: secret "maas-api-e2e-api-b-a955c2-serving-cert" not found
Jul 24 02:07:45.929193 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:07:45.928983    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls podName:a3dbc4e3-7aff-4101-b492-83dde20786ef nodeName:}" failed. No retries permitted until 2026-07-24 02:07:46.428965669 +0000 UTC m=+2286.851032483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls") pod "maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" (UID: "a3dbc4e3-7aff-4101-b492-83dde20786ef") : secret "maas-api-e2e-api-b-a955c2-serving-cert" not found
Jul 24 02:07:45.929573 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.929550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dbc4e3-7aff-4101-b492-83dde20786ef-openshift-service-ca\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:45.939960 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:45.939933    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm2c\" (UniqueName: \"kubernetes.io/projected/a3dbc4e3-7aff-4101-b492-83dde20786ef-kube-api-access-lcm2c\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:46.036875 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:46.036787    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:07:46.434119 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:46.433992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:46.437002 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:46.436969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls\") pod \"maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:46.627159 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:46.627121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:46.782892 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:46.782854    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"]
Jul 24 02:07:46.784465 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:07:46.784441    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dbc4e3_7aff_4101_b492_83dde20786ef.slice/crio-e303dc726ab87b8d58b5bf2762890a0e867112a98a4f58d92d1e852db06f660c WatchSource:0}: Error finding container e303dc726ab87b8d58b5bf2762890a0e867112a98a4f58d92d1e852db06f660c: Status 404 returned error can't find the container with id e303dc726ab87b8d58b5bf2762890a0e867112a98a4f58d92d1e852db06f660c
Jul 24 02:07:47.071295 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:47.071258    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" event={"ID":"a3dbc4e3-7aff-4101-b492-83dde20786ef","Type":"ContainerStarted","Data":"e303dc726ab87b8d58b5bf2762890a0e867112a98a4f58d92d1e852db06f660c"}
Jul 24 02:07:48.079475 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:48.079438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" event={"ID":"a3dbc4e3-7aff-4101-b492-83dde20786ef","Type":"ContainerStarted","Data":"cf63abb04faf545d161091907a11211f0bf9e14f1a1828755f9771ab4b766733"}
Jul 24 02:07:48.079888 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:48.079523    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:48.097869 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:48.097807    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" podStartSLOduration=2.5330676949999997 podStartE2EDuration="3.097789237s" totalImagesPullingTime="564.721542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:07:48.095056336 +0000 UTC m=+2288.517123193" watchObservedRunningTime="2026-07-24 02:07:48.097789237 +0000 UTC m=+2288.519856072"
Jul 24 02:07:50.026448 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:50.026412    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:07:50.026823 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:50.026539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:07:54.091718 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:54.091684    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:07:58.458437 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.458399    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"]
Jul 24 02:07:58.462653 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.462627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:07:58.465218 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.465190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:07:58.470302 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.470270    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"]
Jul 24 02:07:58.556685 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.556636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c78e6b-33c3-4538-8b60-e32429728d23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7\" (UID: \"97c78e6b-33c3-4538-8b60-e32429728d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:07:58.657880 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.657839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c78e6b-33c3-4538-8b60-e32429728d23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7\" (UID: \"97c78e6b-33c3-4538-8b60-e32429728d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:07:58.658591 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.658569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c78e6b-33c3-4538-8b60-e32429728d23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7\" (UID: \"97c78e6b-33c3-4538-8b60-e32429728d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:07:58.775457 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.775372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:07:58.926160 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:58.926126    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"]
Jul 24 02:07:58.931133 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:07:58.931105    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c78e6b_33c3_4538_8b60_e32429728d23.slice/crio-fde7acd15f1e1dc81f7b62b78dc24dd5bf155660d468ac790046457deb11637d WatchSource:0}: Error finding container fde7acd15f1e1dc81f7b62b78dc24dd5bf155660d468ac790046457deb11637d: Status 404 returned error can't find the container with id fde7acd15f1e1dc81f7b62b78dc24dd5bf155660d468ac790046457deb11637d
Jul 24 02:07:59.163878 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:07:59.163844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7" event={"ID":"97c78e6b-33c3-4538-8b60-e32429728d23","Type":"ContainerStarted","Data":"fde7acd15f1e1dc81f7b62b78dc24dd5bf155660d468ac790046457deb11637d"}
Jul 24 02:08:00.172283 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:00.172243    2564 generic.go:356] "Generic (PLEG): container finished" podID="97c78e6b-33c3-4538-8b60-e32429728d23" containerID="cf06dce7067c088618f6eb9a093b4400f319a0f90cd4688cdd0c62c75d6053bf" exitCode=0
Jul 24 02:08:00.172670 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:00.172315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7" event={"ID":"97c78e6b-33c3-4538-8b60-e32429728d23","Type":"ContainerDied","Data":"cf06dce7067c088618f6eb9a093b4400f319a0f90cd4688cdd0c62c75d6053bf"}
Jul 24 02:08:01.324454 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:01.324427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:08:01.381807 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:01.381772    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c78e6b-33c3-4538-8b60-e32429728d23-maas-api-service-ca\") pod \"97c78e6b-33c3-4538-8b60-e32429728d23\" (UID: \"97c78e6b-33c3-4538-8b60-e32429728d23\") "
Jul 24 02:08:01.382204 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:01.382176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c78e6b-33c3-4538-8b60-e32429728d23-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97c78e6b-33c3-4538-8b60-e32429728d23" (UID: "97c78e6b-33c3-4538-8b60-e32429728d23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:08:01.483096 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:01.483065    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c78e6b-33c3-4538-8b60-e32429728d23-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:02.186812 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:02.186781    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"
Jul 24 02:08:02.186967 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:02.186823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7" event={"ID":"97c78e6b-33c3-4538-8b60-e32429728d23","Type":"ContainerDied","Data":"fde7acd15f1e1dc81f7b62b78dc24dd5bf155660d468ac790046457deb11637d"}
Jul 24 02:08:02.186967 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:02.186868    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde7acd15f1e1dc81f7b62b78dc24dd5bf155660d468ac790046457deb11637d"
Jul 24 02:08:14.125437 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.125399    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"]
Jul 24 02:08:14.126038 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.125688    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" podUID="f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" containerName="maas-api" containerID="cri-o://34407157a3c689200462a84eca8c7724f9ce44212de2e1cf248011b05560357a" gracePeriod=30
Jul 24 02:08:14.193900 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.193866    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"]
Jul 24 02:08:14.194372 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.194330    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t" podUID="0c7d9932-6f44-46ca-9a09-726588af7961" containerName="payload-processing" containerID="cri-o://167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce" gracePeriod=30
Jul 24 02:08:14.203573 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.203491    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"]
Jul 24 02:08:14.204251 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.204174    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk" podUID="3233e4e9-b8e7-4c6e-a44d-11fde439549a" containerName="payload-pre-processing" containerID="cri-o://87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3" gracePeriod=30
Jul 24 02:08:14.272492 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.272455    2564 generic.go:356] "Generic (PLEG): container finished" podID="f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" containerID="34407157a3c689200462a84eca8c7724f9ce44212de2e1cf248011b05560357a" exitCode=0
Jul 24 02:08:14.272660 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.272531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" event={"ID":"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc","Type":"ContainerDied","Data":"34407157a3c689200462a84eca8c7724f9ce44212de2e1cf248011b05560357a"}
Jul 24 02:08:14.478794 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.478768    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:08:14.534454 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.534429    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:08:14.540210 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.540179    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:08:14.604518 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkxn6\" (UniqueName: \"kubernetes.io/projected/3233e4e9-b8e7-4c6e-a44d-11fde439549a-kube-api-access-fkxn6\") pod \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") "
Jul 24 02:08:14.604518 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604527    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-openshift-service-ca\") pod \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") "
Jul 24 02:08:14.604826 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znrqr\" (UniqueName: \"kubernetes.io/projected/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-kube-api-access-znrqr\") pod \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") "
Jul 24 02:08:14.604826 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604593    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls\") pod \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\" (UID: \"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc\") "
Jul 24 02:08:14.604826 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604691    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3233e4e9-b8e7-4c6e-a44d-11fde439549a-plugins-config-volume\") pod \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\" (UID: \"3233e4e9-b8e7-4c6e-a44d-11fde439549a\") "
Jul 24 02:08:14.604826 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7d9932-6f44-46ca-9a09-726588af7961-plugins-config-volume\") pod \"0c7d9932-6f44-46ca-9a09-726588af7961\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") "
Jul 24 02:08:14.604826 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.604824    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcfg2\" (UniqueName: \"kubernetes.io/projected/0c7d9932-6f44-46ca-9a09-726588af7961-kube-api-access-kcfg2\") pod \"0c7d9932-6f44-46ca-9a09-726588af7961\" (UID: \"0c7d9932-6f44-46ca-9a09-726588af7961\") "
Jul 24 02:08:14.605119 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.605077    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" (UID: "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:08:14.605186 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.605160    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3233e4e9-b8e7-4c6e-a44d-11fde439549a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3233e4e9-b8e7-4c6e-a44d-11fde439549a" (UID: "3233e4e9-b8e7-4c6e-a44d-11fde439549a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:08:14.605186 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.605171    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7d9932-6f44-46ca-9a09-726588af7961-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c7d9932-6f44-46ca-9a09-726588af7961" (UID: "0c7d9932-6f44-46ca-9a09-726588af7961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:08:14.605317 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.605218    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:14.607439 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.607406    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" (UID: "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:08:14.607439 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.607439    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-kube-api-access-znrqr" (OuterVolumeSpecName: "kube-api-access-znrqr") pod "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" (UID: "f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc"). InnerVolumeSpecName "kube-api-access-znrqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:08:14.607680 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.607523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3233e4e9-b8e7-4c6e-a44d-11fde439549a-kube-api-access-fkxn6" (OuterVolumeSpecName: "kube-api-access-fkxn6") pod "3233e4e9-b8e7-4c6e-a44d-11fde439549a" (UID: "3233e4e9-b8e7-4c6e-a44d-11fde439549a"). InnerVolumeSpecName "kube-api-access-fkxn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:08:14.607680 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.607599    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7d9932-6f44-46ca-9a09-726588af7961-kube-api-access-kcfg2" (OuterVolumeSpecName: "kube-api-access-kcfg2") pod "0c7d9932-6f44-46ca-9a09-726588af7961" (UID: "0c7d9932-6f44-46ca-9a09-726588af7961"). InnerVolumeSpecName "kube-api-access-kcfg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:08:14.705835 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.705803    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7d9932-6f44-46ca-9a09-726588af7961-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:14.705835 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.705834    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcfg2\" (UniqueName: \"kubernetes.io/projected/0c7d9932-6f44-46ca-9a09-726588af7961-kube-api-access-kcfg2\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:14.706069 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.705845    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxn6\" (UniqueName: \"kubernetes.io/projected/3233e4e9-b8e7-4c6e-a44d-11fde439549a-kube-api-access-fkxn6\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:14.706069 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.705854    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrqr\" (UniqueName: \"kubernetes.io/projected/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-kube-api-access-znrqr\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:14.706069 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.705863    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:14.706069 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:14.705872    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3233e4e9-b8e7-4c6e-a44d-11fde439549a-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:15.281322 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.281285    2564 generic.go:356] "Generic (PLEG): container finished" podID="3233e4e9-b8e7-4c6e-a44d-11fde439549a" containerID="87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3" exitCode=0
Jul 24 02:08:15.281322 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.281327    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"
Jul 24 02:08:15.281854 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.281371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk" event={"ID":"3233e4e9-b8e7-4c6e-a44d-11fde439549a","Type":"ContainerDied","Data":"87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3"}
Jul 24 02:08:15.281854 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.281423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk" event={"ID":"3233e4e9-b8e7-4c6e-a44d-11fde439549a","Type":"ContainerDied","Data":"7c72fe8e2af8fba9acf5c9d23e215dadbbd8fae1c400a65ee507a813eddebd5a"}
Jul 24 02:08:15.281854 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.281447    2564 scope.go:117] "RemoveContainer" containerID="87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3"
Jul 24 02:08:15.282891 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.282864    2564 generic.go:356] "Generic (PLEG): container finished" podID="0c7d9932-6f44-46ca-9a09-726588af7961" containerID="167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce" exitCode=0
Jul 24 02:08:15.283047 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.282905    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"
Jul 24 02:08:15.283047 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.282954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t" event={"ID":"0c7d9932-6f44-46ca-9a09-726588af7961","Type":"ContainerDied","Data":"167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce"}
Jul 24 02:08:15.283047 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.282999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t" event={"ID":"0c7d9932-6f44-46ca-9a09-726588af7961","Type":"ContainerDied","Data":"89f4c62064ed44c583b7e4303a0f78ab96b2552b07323c61795a3ee638aa7fdc"}
Jul 24 02:08:15.284561 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.284536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6" event={"ID":"f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc","Type":"ContainerDied","Data":"976825aae7a3c3842916286d2cae3fd2d4be0c07b5c2a770a88f7cab5b2bbb37"}
Jul 24 02:08:15.284682 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.284587    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"
Jul 24 02:08:15.292176 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.292157    2564 scope.go:117] "RemoveContainer" containerID="87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3"
Jul 24 02:08:15.292437 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:08:15.292414    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3\": container with ID starting with 87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3 not found: ID does not exist" containerID="87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3"
Jul 24 02:08:15.292522 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.292447    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3"} err="failed to get container status \"87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3\": rpc error: code = NotFound desc = could not find container \"87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3\": container with ID starting with 87cad2d8dc0ce732da59f3faf95b7eb180df0971feea0b831c326ac95d9d3cd3 not found: ID does not exist"
Jul 24 02:08:15.292522 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.292469    2564 scope.go:117] "RemoveContainer" containerID="167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce"
Jul 24 02:08:15.304441 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.304405    2564 scope.go:117] "RemoveContainer" containerID="167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce"
Jul 24 02:08:15.304903 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:08:15.304884    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce\": container with ID starting with 167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce not found: ID does not exist" containerID="167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce"
Jul 24 02:08:15.304965 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.304910    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce"} err="failed to get container status \"167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce\": rpc error: code = NotFound desc = could not find container \"167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce\": container with ID starting with 167b9cce1a738cc39094c582c0ae49b959f1f9f422392a61df4ee4a8571594ce not found: ID does not exist"
Jul 24 02:08:15.304965 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.304927    2564 scope.go:117] "RemoveContainer" containerID="34407157a3c689200462a84eca8c7724f9ce44212de2e1cf248011b05560357a"
Jul 24 02:08:15.308930 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.308908    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"]
Jul 24 02:08:15.314394 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.314360    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-834f2e-7756f9565b-dqfzk"]
Jul 24 02:08:15.326681 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.326653    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"]
Jul 24 02:08:15.333899 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.333869    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-834f2e-5d5b7f87bc-24vk6"]
Jul 24 02:08:15.346667 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.346635    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"]
Jul 24 02:08:15.350459 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:15.350433    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-834f2e-cc6c76888-t4g9t"]
Jul 24 02:08:16.210465 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:16.210432    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c7d9932-6f44-46ca-9a09-726588af7961" path="/var/lib/kubelet/pods/0c7d9932-6f44-46ca-9a09-726588af7961/volumes"
Jul 24 02:08:16.210831 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:16.210806    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3233e4e9-b8e7-4c6e-a44d-11fde439549a" path="/var/lib/kubelet/pods/3233e4e9-b8e7-4c6e-a44d-11fde439549a/volumes"
Jul 24 02:08:16.211186 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:16.211173    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" path="/var/lib/kubelet/pods/f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc/volumes"
Jul 24 02:08:18.540822 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:18.540785    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"]
Jul 24 02:08:18.543372 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:18.543350    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-834f2e-5zlc7"]
Jul 24 02:08:20.210482 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:20.210446    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c78e6b-33c3-4538-8b60-e32429728d23" path="/var/lib/kubelet/pods/97c78e6b-33c3-4538-8b60-e32429728d23/volumes"
Jul 24 02:08:25.419894 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.419849    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"]
Jul 24 02:08:25.420436 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420397    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3233e4e9-b8e7-4c6e-a44d-11fde439549a" containerName="payload-pre-processing"
Jul 24 02:08:25.420436 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420412    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3233e4e9-b8e7-4c6e-a44d-11fde439549a" containerName="payload-pre-processing"
Jul 24 02:08:25.420436 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420420    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c78e6b-33c3-4538-8b60-e32429728d23" containerName="revoke-keys"
Jul 24 02:08:25.420436 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420426    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c78e6b-33c3-4538-8b60-e32429728d23" containerName="revoke-keys"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420442    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c7d9932-6f44-46ca-9a09-726588af7961" containerName="payload-processing"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420448    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c7d9932-6f44-46ca-9a09-726588af7961" containerName="payload-processing"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420458    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" containerName="maas-api"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420464    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" containerName="maas-api"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420532    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f44ff8-2d57-4ba3-b413-2a3f0bcc76bc" containerName="maas-api"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420540    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c78e6b-33c3-4538-8b60-e32429728d23" containerName="revoke-keys"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420545    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c7d9932-6f44-46ca-9a09-726588af7961" containerName="payload-processing"
Jul 24 02:08:25.420654 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.420553    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3233e4e9-b8e7-4c6e-a44d-11fde439549a" containerName="payload-pre-processing"
Jul 24 02:08:25.423979 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.423953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:25.426587 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.426563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:08:25.433864 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.433840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"]
Jul 24 02:08:25.513835 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.513791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f882625a-cc2e-4fce-9fe1-ef17f399e598-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl\" (UID: \"f882625a-cc2e-4fce-9fe1-ef17f399e598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:25.614480 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.614441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f882625a-cc2e-4fce-9fe1-ef17f399e598-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl\" (UID: \"f882625a-cc2e-4fce-9fe1-ef17f399e598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:25.615155 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.615125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f882625a-cc2e-4fce-9fe1-ef17f399e598-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl\" (UID: \"f882625a-cc2e-4fce-9fe1-ef17f399e598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:25.736659 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.736608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:25.889606 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:25.889572    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"]
Jul 24 02:08:25.891365 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:08:25.891335    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf882625a_cc2e_4fce_9fe1_ef17f399e598.slice/crio-d37956762ceaa2604b8f04cebbb7fb3bc7784b3950f98df8eab537d5c7c138eb WatchSource:0}: Error finding container d37956762ceaa2604b8f04cebbb7fb3bc7784b3950f98df8eab537d5c7c138eb: Status 404 returned error can't find the container with id d37956762ceaa2604b8f04cebbb7fb3bc7784b3950f98df8eab537d5c7c138eb
Jul 24 02:08:26.361239 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:26.361204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl" event={"ID":"f882625a-cc2e-4fce-9fe1-ef17f399e598","Type":"ContainerStarted","Data":"3a3633a42491aa895bf2a522f10b79d748450b8e689672d6839a67e6916539ca"}
Jul 24 02:08:26.361417 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:26.361260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl" event={"ID":"f882625a-cc2e-4fce-9fe1-ef17f399e598","Type":"ContainerStarted","Data":"d37956762ceaa2604b8f04cebbb7fb3bc7784b3950f98df8eab537d5c7c138eb"}
Jul 24 02:08:26.377905 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:26.377795    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl" podStartSLOduration=1.377777381 podStartE2EDuration="1.377777381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:08:26.37546452 +0000 UTC m=+2326.797531357" watchObservedRunningTime="2026-07-24 02:08:26.377777381 +0000 UTC m=+2326.799844270"
Jul 24 02:08:27.370408 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:27.370369    2564 generic.go:356] "Generic (PLEG): container finished" podID="f882625a-cc2e-4fce-9fe1-ef17f399e598" containerID="3a3633a42491aa895bf2a522f10b79d748450b8e689672d6839a67e6916539ca" exitCode=0
Jul 24 02:08:27.370878 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:27.370440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl" event={"ID":"f882625a-cc2e-4fce-9fe1-ef17f399e598","Type":"ContainerDied","Data":"3a3633a42491aa895bf2a522f10b79d748450b8e689672d6839a67e6916539ca"}
Jul 24 02:08:28.516373 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:28.516345    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:28.644864 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:28.644759    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f882625a-cc2e-4fce-9fe1-ef17f399e598-maas-api-service-ca\") pod \"f882625a-cc2e-4fce-9fe1-ef17f399e598\" (UID: \"f882625a-cc2e-4fce-9fe1-ef17f399e598\") "
Jul 24 02:08:28.645240 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:28.645207    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f882625a-cc2e-4fce-9fe1-ef17f399e598-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f882625a-cc2e-4fce-9fe1-ef17f399e598" (UID: "f882625a-cc2e-4fce-9fe1-ef17f399e598"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:08:28.745813 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:28.745777    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f882625a-cc2e-4fce-9fe1-ef17f399e598-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:29.383514 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:29.383478    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"
Jul 24 02:08:29.383514 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:29.383503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl" event={"ID":"f882625a-cc2e-4fce-9fe1-ef17f399e598","Type":"ContainerDied","Data":"d37956762ceaa2604b8f04cebbb7fb3bc7784b3950f98df8eab537d5c7c138eb"}
Jul 24 02:08:29.383749 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:29.383565    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37956762ceaa2604b8f04cebbb7fb3bc7784b3950f98df8eab537d5c7c138eb"
Jul 24 02:08:41.282021 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.281968    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"]
Jul 24 02:08:41.284466 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.282305    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" podUID="a3dbc4e3-7aff-4101-b492-83dde20786ef" containerName="maas-api" containerID="cri-o://cf63abb04faf545d161091907a11211f0bf9e14f1a1828755f9771ab4b766733" gracePeriod=30
Jul 24 02:08:41.462535 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.462500    2564 generic.go:356] "Generic (PLEG): container finished" podID="a3dbc4e3-7aff-4101-b492-83dde20786ef" containerID="cf63abb04faf545d161091907a11211f0bf9e14f1a1828755f9771ab4b766733" exitCode=0
Jul 24 02:08:41.462708 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.462531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" event={"ID":"a3dbc4e3-7aff-4101-b492-83dde20786ef","Type":"ContainerDied","Data":"cf63abb04faf545d161091907a11211f0bf9e14f1a1828755f9771ab4b766733"}
Jul 24 02:08:41.566249 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.566218    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:08:41.573062 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.573008    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls\") pod \"a3dbc4e3-7aff-4101-b492-83dde20786ef\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") "
Jul 24 02:08:41.573219 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.573095    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcm2c\" (UniqueName: \"kubernetes.io/projected/a3dbc4e3-7aff-4101-b492-83dde20786ef-kube-api-access-lcm2c\") pod \"a3dbc4e3-7aff-4101-b492-83dde20786ef\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") "
Jul 24 02:08:41.573260 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.573240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dbc4e3-7aff-4101-b492-83dde20786ef-openshift-service-ca\") pod \"a3dbc4e3-7aff-4101-b492-83dde20786ef\" (UID: \"a3dbc4e3-7aff-4101-b492-83dde20786ef\") "
Jul 24 02:08:41.573596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.573572    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dbc4e3-7aff-4101-b492-83dde20786ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3dbc4e3-7aff-4101-b492-83dde20786ef" (UID: "a3dbc4e3-7aff-4101-b492-83dde20786ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:08:41.575217 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.575189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3dbc4e3-7aff-4101-b492-83dde20786ef" (UID: "a3dbc4e3-7aff-4101-b492-83dde20786ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:08:41.575338 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.575228    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dbc4e3-7aff-4101-b492-83dde20786ef-kube-api-access-lcm2c" (OuterVolumeSpecName: "kube-api-access-lcm2c") pod "a3dbc4e3-7aff-4101-b492-83dde20786ef" (UID: "a3dbc4e3-7aff-4101-b492-83dde20786ef"). InnerVolumeSpecName "kube-api-access-lcm2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:08:41.674411 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.674375    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dbc4e3-7aff-4101-b492-83dde20786ef-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:41.674411 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.674406    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dbc4e3-7aff-4101-b492-83dde20786ef-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:41.674411 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:41.674416    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcm2c\" (UniqueName: \"kubernetes.io/projected/a3dbc4e3-7aff-4101-b492-83dde20786ef-kube-api-access-lcm2c\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:08:42.228148 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:42.228116    2564 scope.go:117] "RemoveContainer" containerID="be7682a9896cc36c90ca3415dc8767ccec4d5e96aa519747e551586501187807"
Jul 24 02:08:42.470516 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:42.470432    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"
Jul 24 02:08:42.470516 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:42.470452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q" event={"ID":"a3dbc4e3-7aff-4101-b492-83dde20786ef","Type":"ContainerDied","Data":"e303dc726ab87b8d58b5bf2762890a0e867112a98a4f58d92d1e852db06f660c"}
Jul 24 02:08:42.471379 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:42.470516    2564 scope.go:117] "RemoveContainer" containerID="cf63abb04faf545d161091907a11211f0bf9e14f1a1828755f9771ab4b766733"
Jul 24 02:08:42.488613 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:42.488565    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"]
Jul 24 02:08:42.492285 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:42.492251    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a955c2-55d7b5866c-brb4q"]
Jul 24 02:08:44.210287 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:44.210248    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3dbc4e3-7aff-4101-b492-83dde20786ef" path="/var/lib/kubelet/pods/a3dbc4e3-7aff-4101-b492-83dde20786ef/volumes"
Jul 24 02:08:45.522095 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:45.522058    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"]
Jul 24 02:08:45.526679 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:45.526654    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a955c2-cpjhl"]
Jul 24 02:08:46.210719 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:46.210683    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f882625a-cc2e-4fce-9fe1-ef17f399e598" path="/var/lib/kubelet/pods/f882625a-cc2e-4fce-9fe1-ef17f399e598/volumes"
Jul 24 02:08:53.056108 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056074    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"]
Jul 24 02:08:53.056617 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056587    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3dbc4e3-7aff-4101-b492-83dde20786ef" containerName="maas-api"
Jul 24 02:08:53.056662 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056619    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3dbc4e3-7aff-4101-b492-83dde20786ef" containerName="maas-api"
Jul 24 02:08:53.056662 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056643    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f882625a-cc2e-4fce-9fe1-ef17f399e598" containerName="revoke-keys"
Jul 24 02:08:53.056662 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056649    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f882625a-cc2e-4fce-9fe1-ef17f399e598" containerName="revoke-keys"
Jul 24 02:08:53.056755 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056713    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3dbc4e3-7aff-4101-b492-83dde20786ef" containerName="maas-api"
Jul 24 02:08:53.056755 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.056723    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f882625a-cc2e-4fce-9fe1-ef17f399e598" containerName="revoke-keys"
Jul 24 02:08:53.059995 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.059974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.065110 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.065083    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6bcabd-openshift-default-dockercfg-z9tmn\""
Jul 24 02:08:53.079621 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.079590    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"]
Jul 24 02:08:53.190015 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.189979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-socket\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190015 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190016    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-envoy\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190302 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-certs\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190302 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-credential-socket\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190302 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-token\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190468 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istiod-ca-cert\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190468 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-podinfo\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190468 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190402    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2dk\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-kube-api-access-xf2dk\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.190468 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.190443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-data\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.291861 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.291823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-certs\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.291879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-credential-socket\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.291901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-token\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.291936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istiod-ca-cert\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.291959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-podinfo\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2dk\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-kube-api-access-xf2dk\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-data\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292121 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-socket\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-envoy\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-certs\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292437    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-credential-socket\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292627 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-socket\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292808 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-data\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.292846 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.292802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istiod-ca-cert\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.294533 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.294512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-envoy\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.294669 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.294650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-podinfo\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.301099 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.301067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-token\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.301229 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.301124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2dk\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-kube-api-access-xf2dk\") pod \"e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") " pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.370977 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.370885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:53.522837 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.522810    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"]
Jul 24 02:08:53.524449 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:08:53.524419    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0329af3_a7fe_48bd_a856_c287c6c11fd5.slice/crio-8e6986e055d8a668e5350eaf84aa989b25cf77fc03c2871f6b6fa5927855d023 WatchSource:0}: Error finding container 8e6986e055d8a668e5350eaf84aa989b25cf77fc03c2871f6b6fa5927855d023: Status 404 returned error can't find the container with id 8e6986e055d8a668e5350eaf84aa989b25cf77fc03c2871f6b6fa5927855d023
Jul 24 02:08:53.526664 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.526632    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:08:53.526740 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.526699    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:08:53.526778 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.526748    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:08:53.538709 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:53.538684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" event={"ID":"f0329af3-a7fe-48bd-a856-c287c6c11fd5","Type":"ContainerStarted","Data":"8e6986e055d8a668e5350eaf84aa989b25cf77fc03c2871f6b6fa5927855d023"}
Jul 24 02:08:54.547475 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:54.547439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" event={"ID":"f0329af3-a7fe-48bd-a856-c287c6c11fd5","Type":"ContainerStarted","Data":"802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"}
Jul 24 02:08:54.570544 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:54.570485    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podStartSLOduration=1.570469398 podStartE2EDuration="1.570469398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:08:54.567296732 +0000 UTC m=+2354.989363616" watchObservedRunningTime="2026-07-24 02:08:54.570469398 +0000 UTC m=+2354.992536234"
Jul 24 02:08:55.371403 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:55.371368    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:55.376431 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:55.376405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:55.553851 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:55.553820    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:55.555005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:55.554989    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:08:56.673851 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.673813    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"]
Jul 24 02:08:56.677726 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.677704    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.679957 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.679933    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6bcabd-serving-cert\""
Jul 24 02:08:56.690167 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.690142    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"]
Jul 24 02:08:56.730907 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.730873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.731146 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.730932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2mr2\" (UniqueName: \"kubernetes.io/projected/a2aa36a1-5350-4d93-9721-6c24562a36e5-kube-api-access-w2mr2\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.731146 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.731008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa36a1-5350-4d93-9721-6c24562a36e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.832603 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.832570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.832806 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.832620    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2mr2\" (UniqueName: \"kubernetes.io/projected/a2aa36a1-5350-4d93-9721-6c24562a36e5-kube-api-access-w2mr2\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.832806 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.832658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa36a1-5350-4d93-9721-6c24562a36e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.832806 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:08:56.832746    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-serving-cert: secret "maas-api-e2e-shared-a-6bcabd-serving-cert" not found
Jul 24 02:08:56.832972 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:08:56.832827    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls podName:a2aa36a1-5350-4d93-9721-6c24562a36e5 nodeName:}" failed. No retries permitted until 2026-07-24 02:08:57.332806874 +0000 UTC m=+2357.754873701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls") pod "maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" (UID: "a2aa36a1-5350-4d93-9721-6c24562a36e5") : secret "maas-api-e2e-shared-a-6bcabd-serving-cert" not found
Jul 24 02:08:56.833341 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.833324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa36a1-5350-4d93-9721-6c24562a36e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:56.844791 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:56.844768    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2mr2\" (UniqueName: \"kubernetes.io/projected/a2aa36a1-5350-4d93-9721-6c24562a36e5-kube-api-access-w2mr2\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:57.338244 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:57.338204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:57.340848 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:57.340821    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:57.590812 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:57.590709    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:57.753535 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:57.753505    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"]
Jul 24 02:08:57.755755 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:08:57.755726    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2aa36a1_5350_4d93_9721_6c24562a36e5.slice/crio-a4002d4cc6ceaf0fc3bcc74d2f7b3b6cf0c49e3bea09fda540c1f2c84285b7a2 WatchSource:0}: Error finding container a4002d4cc6ceaf0fc3bcc74d2f7b3b6cf0c49e3bea09fda540c1f2c84285b7a2: Status 404 returned error can't find the container with id a4002d4cc6ceaf0fc3bcc74d2f7b3b6cf0c49e3bea09fda540c1f2c84285b7a2
Jul 24 02:08:58.576074 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:58.576015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" event={"ID":"a2aa36a1-5350-4d93-9721-6c24562a36e5","Type":"ContainerStarted","Data":"638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0"}
Jul 24 02:08:58.576254 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:58.576091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" event={"ID":"a2aa36a1-5350-4d93-9721-6c24562a36e5","Type":"ContainerStarted","Data":"a4002d4cc6ceaf0fc3bcc74d2f7b3b6cf0c49e3bea09fda540c1f2c84285b7a2"}
Jul 24 02:08:58.576254 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:58.576244    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:08:58.599825 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:08:58.599768    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" podStartSLOduration=1.953852934 podStartE2EDuration="2.59975075s" totalImagesPullingTime="645.897816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:08:58.598953972 +0000 UTC m=+2359.021020820" watchObservedRunningTime="2026-07-24 02:08:58.59975075 +0000 UTC m=+2359.021817586"
Jul 24 02:09:04.588433 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:04.588402    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:09:05.086472 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.086433    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"]
Jul 24 02:09:05.090766 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.090740    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.093582 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.093552    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a7fbc1-serving-cert\""
Jul 24 02:09:05.108983 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.108947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"]
Jul 24 02:09:05.227668 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.227615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8qz\" (UniqueName: \"kubernetes.io/projected/825cabe5-6b08-492e-ba96-497335fb0cb8-kube-api-access-dl8qz\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.227989 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.227965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.228251 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.228227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825cabe5-6b08-492e-ba96-497335fb0cb8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.328999 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.328945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.329276 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.329251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825cabe5-6b08-492e-ba96-497335fb0cb8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.329398 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.329378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8qz\" (UniqueName: \"kubernetes.io/projected/825cabe5-6b08-492e-ba96-497335fb0cb8-kube-api-access-dl8qz\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.329887 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:09:05.329858    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-serving-cert: secret "maas-api-e2e-shared-b-a7fbc1-serving-cert" not found
Jul 24 02:09:05.329999 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:09:05.329935    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls podName:825cabe5-6b08-492e-ba96-497335fb0cb8 nodeName:}" failed. No retries permitted until 2026-07-24 02:09:05.829913508 +0000 UTC m=+2366.251980328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls") pod "maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" (UID: "825cabe5-6b08-492e-ba96-497335fb0cb8") : secret "maas-api-e2e-shared-b-a7fbc1-serving-cert" not found
Jul 24 02:09:05.330898 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.330871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825cabe5-6b08-492e-ba96-497335fb0cb8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.340226 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.340150    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8qz\" (UniqueName: \"kubernetes.io/projected/825cabe5-6b08-492e-ba96-497335fb0cb8-kube-api-access-dl8qz\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.837730 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:05.837688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:05.838269 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:09:05.837832    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-serving-cert: secret "maas-api-e2e-shared-b-a7fbc1-serving-cert" not found
Jul 24 02:09:05.838269 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:09:05.837916    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls podName:825cabe5-6b08-492e-ba96-497335fb0cb8 nodeName:}" failed. No retries permitted until 2026-07-24 02:09:06.837895672 +0000 UTC m=+2367.259962496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls") pod "maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" (UID: "825cabe5-6b08-492e-ba96-497335fb0cb8") : secret "maas-api-e2e-shared-b-a7fbc1-serving-cert" not found
Jul 24 02:09:06.850096 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:06.850049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:06.852548 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:06.852526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") pod \"maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:06.903183 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:06.903143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:07.091939 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:07.091911    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"]
Jul 24 02:09:07.093948 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:09:07.093915    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825cabe5_6b08_492e_ba96_497335fb0cb8.slice/crio-04283bd3d59cb1697478abcb3d5f01bf5580fd42a4559a98f18d28a03b6c94d9 WatchSource:0}: Error finding container 04283bd3d59cb1697478abcb3d5f01bf5580fd42a4559a98f18d28a03b6c94d9: Status 404 returned error can't find the container with id 04283bd3d59cb1697478abcb3d5f01bf5580fd42a4559a98f18d28a03b6c94d9
Jul 24 02:09:07.648090 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:07.648060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" event={"ID":"825cabe5-6b08-492e-ba96-497335fb0cb8","Type":"ContainerStarted","Data":"04283bd3d59cb1697478abcb3d5f01bf5580fd42a4559a98f18d28a03b6c94d9"}
Jul 24 02:09:08.655907 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:08.655868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" event={"ID":"825cabe5-6b08-492e-ba96-497335fb0cb8","Type":"ContainerStarted","Data":"17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971"}
Jul 24 02:09:08.656353 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:08.655930    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:08.678088 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:08.678017    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" podStartSLOduration=3.218625182 podStartE2EDuration="3.678002546s" totalImagesPullingTime="459.377364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:09:08.67619424 +0000 UTC m=+2369.098261112" watchObservedRunningTime="2026-07-24 02:09:08.678002546 +0000 UTC m=+2369.100069381"
Jul 24 02:09:11.595281 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:11.595237    2564 patch_prober.go:28] interesting pod/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 02:09:11.595967 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:11.595922    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 02:09:12.684233 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:12.684201    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerID="802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f" exitCode=0
Jul 24 02:09:12.684695 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:12.684279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" event={"ID":"f0329af3-a7fe-48bd-a856-c287c6c11fd5","Type":"ContainerDied","Data":"802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"}
Jul 24 02:09:12.684695 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:12.684692    2564 scope.go:117] "RemoveContainer" containerID="802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"
Jul 24 02:09:12.685765 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:12.685735    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:09:12.685868 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:12.685801    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:09:12.685868 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:12.685832    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:09:13.692119 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:13.692083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" event={"ID":"f0329af3-a7fe-48bd-a856-c287c6c11fd5","Type":"ContainerStarted","Data":"208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741"}
Jul 24 02:09:14.371593 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:14.371557    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:09:14.371876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:14.371853    2564 patch_prober.go:28] interesting pod/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 24 02:09:14.371972 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:14.371910    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 24 02:09:14.668759 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:14.668680    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:09:15.371302 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:15.371266    2564 patch_prober.go:28] interesting pod/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 24 02:09:15.371718 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:15.371319    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 24 02:09:16.372216 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:16.372176    2564 patch_prober.go:28] interesting pod/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 24 02:09:16.372742 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:16.372245    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 24 02:09:17.371474 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:17.371433    2564 patch_prober.go:28] interesting pod/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 24 02:09:17.371753 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:17.371498    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 24 02:09:18.375539 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:18.375505    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:09:18.376078 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:18.376051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:09:18.376781 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:18.376764    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:09:40.584302 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:40.584269    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:09:40.591344 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:40.591311    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:09:40.598035 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:40.597996    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:09:40.604610 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:40.604592    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:09:42.266143 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:09:42.266113    2564 scope.go:117] "RemoveContainer" containerID="7474280fdd19f39597b26185c5193b452347eef6780b40e9697f85908e274c9c"
Jul 24 02:11:42.304546 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:11:42.304515    2564 scope.go:117] "RemoveContainer" containerID="e0216c577f5e42a61a48471bd773c6b9d0f03f76a9d594e5e5af0e83fad7fe7f"
Jul 24 02:13:42.337803 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:13:42.337764    2564 scope.go:117] "RemoveContainer" containerID="31ec673f441abce75a7fa47c190306165c7157f65024b51fec566199805f84f8"
Jul 24 02:13:42.347955 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:13:42.347929    2564 scope.go:117] "RemoveContainer" containerID="4197618a4f18635b350d956712611f6c34d53a555f0d87b870b914711b371f99"
Jul 24 02:14:40.626878 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:14:40.626756    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:14:40.634699 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:14:40.634673    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:14:40.643773 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:14:40.643750    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:14:40.650366 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:14:40.650347    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:14:42.376432 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:14:42.376396    2564 scope.go:117] "RemoveContainer" containerID="3a3633a42491aa895bf2a522f10b79d748450b8e689672d6839a67e6916539ca"
Jul 24 02:14:42.388787 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:14:42.388765    2564 scope.go:117] "RemoveContainer" containerID="cf06dce7067c088618f6eb9a093b4400f319a0f90cd4688cdd0c62c75d6053bf"
Jul 24 02:15:00.141496 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.141463    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"]
Jul 24 02:15:00.145512 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.145487    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"]
Jul 24 02:15:00.145672 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.145609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"
Jul 24 02:15:00.148343 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.148276    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:15:00.149376 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.149350    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"]
Jul 24 02:15:00.149583 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.149448    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"
Jul 24 02:15:00.155481 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.155458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"
Jul 24 02:15:00.155729 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.155364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"]
Jul 24 02:15:00.157694 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.157537    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"]
Jul 24 02:15:00.160236 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.160213    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"]
Jul 24 02:15:00.165593 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.165569    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"
Jul 24 02:15:00.174773 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.174750    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"
Jul 24 02:15:00.213977 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.213953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"
Jul 24 02:15:00.346634 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.346606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"]
Jul 24 02:15:00.348852 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:15:00.348824    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8587860_e3b4_4d55_bafe_9c7978758b71.slice/crio-bbc83043e58e88075bb7f342a38527279af653181bdddf4bdc4a7f6ab2cbbdf2 WatchSource:0}: Error finding container bbc83043e58e88075bb7f342a38527279af653181bdddf4bdc4a7f6ab2cbbdf2: Status 404 returned error can't find the container with id bbc83043e58e88075bb7f342a38527279af653181bdddf4bdc4a7f6ab2cbbdf2
Jul 24 02:15:00.406002 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.405974    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"]
Jul 24 02:15:00.407055 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:15:00.407008    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a4fe33_d9f4_4d15_a7c2_124937df54d7.slice/crio-f6da3bd6cb9e8b130e9c6c7bdd502c0114ddec3f22d4251041901dc4a9234ffe WatchSource:0}: Error finding container f6da3bd6cb9e8b130e9c6c7bdd502c0114ddec3f22d4251041901dc4a9234ffe: Status 404 returned error can't find the container with id f6da3bd6cb9e8b130e9c6c7bdd502c0114ddec3f22d4251041901dc4a9234ffe
Jul 24 02:15:00.587492 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.587458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"]
Jul 24 02:15:00.589398 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:15:00.589371    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode350cac9_965b_4d77_9964_32af3e21675b.slice/crio-3d9006c631e8e548a960b95312c7a95be0579eb55210f3cf3ab3e0e4b31de56a WatchSource:0}: Error finding container 3d9006c631e8e548a960b95312c7a95be0579eb55210f3cf3ab3e0e4b31de56a: Status 404 returned error can't find the container with id 3d9006c631e8e548a960b95312c7a95be0579eb55210f3cf3ab3e0e4b31de56a
Jul 24 02:15:00.899010 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.898968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5" event={"ID":"e350cac9-965b-4d77-9964-32af3e21675b","Type":"ContainerStarted","Data":"3d9006c631e8e548a960b95312c7a95be0579eb55210f3cf3ab3e0e4b31de56a"}
Jul 24 02:15:00.900639 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.900614    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc" event={"ID":"e0a4fe33-d9f4-4d15-a7c2-124937df54d7","Type":"ContainerStarted","Data":"4853b8db3a51c15028b42e29b4fd6111c055303dc366db30be2b3bea5706f238"}
Jul 24 02:15:00.900807 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.900655    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc" event={"ID":"e0a4fe33-d9f4-4d15-a7c2-124937df54d7","Type":"ContainerStarted","Data":"f6da3bd6cb9e8b130e9c6c7bdd502c0114ddec3f22d4251041901dc4a9234ffe"}
Jul 24 02:15:00.902290 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.902270    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll" event={"ID":"b8587860-e3b4-4d55-bafe-9c7978758b71","Type":"ContainerStarted","Data":"f2a02f7588d4cdf171c01d4fcf8307dd616e9f7fbf5c2b3724a9f04e59240d83"}
Jul 24 02:15:00.902409 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.902306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll" event={"ID":"b8587860-e3b4-4d55-bafe-9c7978758b71","Type":"ContainerStarted","Data":"bbc83043e58e88075bb7f342a38527279af653181bdddf4bdc4a7f6ab2cbbdf2"}
Jul 24 02:15:00.919168 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.919117    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc" podStartSLOduration=0.919099166 podStartE2EDuration="919.099166ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:15:00.916486015 +0000 UTC m=+2721.338552850" watchObservedRunningTime="2026-07-24 02:15:00.919099166 +0000 UTC m=+2721.341166001"
Jul 24 02:15:00.935356 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:00.935298    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll" podStartSLOduration=0.935284187 podStartE2EDuration="935.284187ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:15:00.932791273 +0000 UTC m=+2721.354858130" watchObservedRunningTime="2026-07-24 02:15:00.935284187 +0000 UTC m=+2721.357351039"
Jul 24 02:15:01.915463 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:01.915356    2564 generic.go:356] "Generic (PLEG): container finished" podID="e0a4fe33-d9f4-4d15-a7c2-124937df54d7" containerID="4853b8db3a51c15028b42e29b4fd6111c055303dc366db30be2b3bea5706f238" exitCode=0
Jul 24 02:15:01.915463 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:01.915440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc" event={"ID":"e0a4fe33-d9f4-4d15-a7c2-124937df54d7","Type":"ContainerDied","Data":"4853b8db3a51c15028b42e29b4fd6111c055303dc366db30be2b3bea5706f238"}
Jul 24 02:15:01.917259 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:01.917236    2564 generic.go:356] "Generic (PLEG): container finished" podID="b8587860-e3b4-4d55-bafe-9c7978758b71" containerID="f2a02f7588d4cdf171c01d4fcf8307dd616e9f7fbf5c2b3724a9f04e59240d83" exitCode=0
Jul 24 02:15:01.917365 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:01.917330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll" event={"ID":"b8587860-e3b4-4d55-bafe-9c7978758b71","Type":"ContainerDied","Data":"f2a02f7588d4cdf171c01d4fcf8307dd616e9f7fbf5c2b3724a9f04e59240d83"}
Jul 24 02:15:01.918879 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:01.918859    2564 generic.go:356] "Generic (PLEG): container finished" podID="e350cac9-965b-4d77-9964-32af3e21675b" containerID="631560ced9e3c8f19b39f930009015b9081f0d384f5b101caed814790ac1cde0" exitCode=0
Jul 24 02:15:01.918998 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:01.918887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5" event={"ID":"e350cac9-965b-4d77-9964-32af3e21675b","Type":"ContainerDied","Data":"631560ced9e3c8f19b39f930009015b9081f0d384f5b101caed814790ac1cde0"}
Jul 24 02:15:03.074464 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.074434    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"
Jul 24 02:15:03.138617 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.138589    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"
Jul 24 02:15:03.142155 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.142130    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"
Jul 24 02:15:03.934159 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.934127    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5"
Jul 24 02:15:03.934159 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.934143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747655-srjh5" event={"ID":"e350cac9-965b-4d77-9964-32af3e21675b","Type":"ContainerDied","Data":"3d9006c631e8e548a960b95312c7a95be0579eb55210f3cf3ab3e0e4b31de56a"}
Jul 24 02:15:03.934409 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.934206    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d9006c631e8e548a960b95312c7a95be0579eb55210f3cf3ab3e0e4b31de56a"
Jul 24 02:15:03.935706 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.935685    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"
Jul 24 02:15:03.935706 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.935691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc" event={"ID":"e0a4fe33-d9f4-4d15-a7c2-124937df54d7","Type":"ContainerDied","Data":"f6da3bd6cb9e8b130e9c6c7bdd502c0114ddec3f22d4251041901dc4a9234ffe"}
Jul 24 02:15:03.935901 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.935730    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6da3bd6cb9e8b130e9c6c7bdd502c0114ddec3f22d4251041901dc4a9234ffe"
Jul 24 02:15:03.937424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.937318    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"
Jul 24 02:15:03.937424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.937343    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll" event={"ID":"b8587860-e3b4-4d55-bafe-9c7978758b71","Type":"ContainerDied","Data":"bbc83043e58e88075bb7f342a38527279af653181bdddf4bdc4a7f6ab2cbbdf2"}
Jul 24 02:15:03.937424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:03.937383    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc83043e58e88075bb7f342a38527279af653181bdddf4bdc4a7f6ab2cbbdf2"
Jul 24 02:15:31.222487 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.222451    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"]
Jul 24 02:15:31.223176 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223156    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e350cac9-965b-4d77-9964-32af3e21675b" containerName="cleanup"
Jul 24 02:15:31.223243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223179    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e350cac9-965b-4d77-9964-32af3e21675b" containerName="cleanup"
Jul 24 02:15:31.223243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223198    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a4fe33-d9f4-4d15-a7c2-124937df54d7" containerName="cleanup"
Jul 24 02:15:31.223243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223207    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a4fe33-d9f4-4d15-a7c2-124937df54d7" containerName="cleanup"
Jul 24 02:15:31.223243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223234    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8587860-e3b4-4d55-bafe-9c7978758b71" containerName="cleanup"
Jul 24 02:15:31.223243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223243    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8587860-e3b4-4d55-bafe-9c7978758b71" containerName="cleanup"
Jul 24 02:15:31.223501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223370    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e350cac9-965b-4d77-9964-32af3e21675b" containerName="cleanup"
Jul 24 02:15:31.223501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223385    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8587860-e3b4-4d55-bafe-9c7978758b71" containerName="cleanup"
Jul 24 02:15:31.223501 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.223395    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a4fe33-d9f4-4d15-a7c2-124937df54d7" containerName="cleanup"
Jul 24 02:15:31.226565 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.226544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.228927 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.228904    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a002f7-serving-cert\""
Jul 24 02:15:31.238952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.238926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"]
Jul 24 02:15:31.392201 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.392165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63509298-50c1-43db-b47f-52137811bcb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.392410 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.392335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5nq\" (UniqueName: \"kubernetes.io/projected/63509298-50c1-43db-b47f-52137811bcb8-kube-api-access-qn5nq\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.392410 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.392372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63509298-50c1-43db-b47f-52137811bcb8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.493833 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.493751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5nq\" (UniqueName: \"kubernetes.io/projected/63509298-50c1-43db-b47f-52137811bcb8-kube-api-access-qn5nq\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.493833 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.493785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63509298-50c1-43db-b47f-52137811bcb8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.493833 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.493829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63509298-50c1-43db-b47f-52137811bcb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.494544 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.494518    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63509298-50c1-43db-b47f-52137811bcb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.496463 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.496435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63509298-50c1-43db-b47f-52137811bcb8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.502324 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.502302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5nq\" (UniqueName: \"kubernetes.io/projected/63509298-50c1-43db-b47f-52137811bcb8-kube-api-access-qn5nq\") pod \"maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.539455 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.539428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:31.694228 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.693315    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"]
Jul 24 02:15:31.698209 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.695507    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 02:15:31.850092 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.850001    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"]
Jul 24 02:15:31.855215 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.855195    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:31.857849 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.857827    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a002f7\""
Jul 24 02:15:31.865761 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.865734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"]
Jul 24 02:15:31.999463 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.999424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324359aa-48e2-40d4-8dec-724224295a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:31.999762 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:31.999740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6cn\" (UniqueName: \"kubernetes.io/projected/324359aa-48e2-40d4-8dec-724224295a4f-kube-api-access-9j6cn\") pod \"payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:32.101225 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.101136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6cn\" (UniqueName: \"kubernetes.io/projected/324359aa-48e2-40d4-8dec-724224295a4f-kube-api-access-9j6cn\") pod \"payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:32.101225 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.101206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324359aa-48e2-40d4-8dec-724224295a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:32.101848 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.101829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324359aa-48e2-40d4-8dec-724224295a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:32.108762 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.108728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p" event={"ID":"63509298-50c1-43db-b47f-52137811bcb8","Type":"ContainerStarted","Data":"324dc7e7e132dd71313688fb39d0f007158ed217216139da105f94da61aea90d"}
Jul 24 02:15:32.109500 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.109477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6cn\" (UniqueName: \"kubernetes.io/projected/324359aa-48e2-40d4-8dec-724224295a4f-kube-api-access-9j6cn\") pod \"payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:32.169352 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.169317    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:32.529332 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:32.529297    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"]
Jul 24 02:15:32.530776 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:15:32.530750    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324359aa_48e2_40d4_8dec_724224295a4f.slice/crio-3d7fb6a92cd7e87a0845431a8a66408f44c175eac2966820ad8f9e8cc93cd427 WatchSource:0}: Error finding container 3d7fb6a92cd7e87a0845431a8a66408f44c175eac2966820ad8f9e8cc93cd427: Status 404 returned error can't find the container with id 3d7fb6a92cd7e87a0845431a8a66408f44c175eac2966820ad8f9e8cc93cd427
Jul 24 02:15:33.118467 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.118416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p" event={"ID":"63509298-50c1-43db-b47f-52137811bcb8","Type":"ContainerStarted","Data":"34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6"}
Jul 24 02:15:33.118683 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.118486    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:33.120215 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.120192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d" event={"ID":"324359aa-48e2-40d4-8dec-724224295a4f","Type":"ContainerStarted","Data":"92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6"}
Jul 24 02:15:33.120354 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.120230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d" event={"ID":"324359aa-48e2-40d4-8dec-724224295a4f","Type":"ContainerStarted","Data":"3d7fb6a92cd7e87a0845431a8a66408f44c175eac2966820ad8f9e8cc93cd427"}
Jul 24 02:15:33.120354 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.120278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:33.137516 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.137466    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p" podStartSLOduration=1.588758453 podStartE2EDuration="2.137450077s" totalImagesPullingTime="548.691624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:15:33.134428314 +0000 UTC m=+2753.556495173" watchObservedRunningTime="2026-07-24 02:15:33.137450077 +0000 UTC m=+2753.559516913"
Jul 24 02:15:33.149210 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:33.149156    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d" podStartSLOduration=2.149137475 podStartE2EDuration="2.149137475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:15:33.148539477 +0000 UTC m=+2753.570606334" watchObservedRunningTime="2026-07-24 02:15:33.149137475 +0000 UTC m=+2753.571204316"
Jul 24 02:15:39.137186 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:39.137145    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:15:44.133215 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:44.133179    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:15:50.783695 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.783655    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"]
Jul 24 02:15:50.787367 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.787351    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:50.789682 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.789664    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-477d74-serving-cert\""
Jul 24 02:15:50.797189 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.797165    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"]
Jul 24 02:15:50.903566 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.903527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a93108-e4b5-41af-ae1a-7c203fd82965-maas-api-tls\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:50.903735 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.903604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a93108-e4b5-41af-ae1a-7c203fd82965-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:50.903735 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:50.903652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr9s\" (UniqueName: \"kubernetes.io/projected/06a93108-e4b5-41af-ae1a-7c203fd82965-kube-api-access-7nr9s\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.004436 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.004403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a93108-e4b5-41af-ae1a-7c203fd82965-maas-api-tls\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.004627 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.004471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a93108-e4b5-41af-ae1a-7c203fd82965-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.004627 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.004499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr9s\" (UniqueName: \"kubernetes.io/projected/06a93108-e4b5-41af-ae1a-7c203fd82965-kube-api-access-7nr9s\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.005264 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.005234    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a93108-e4b5-41af-ae1a-7c203fd82965-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.006951 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.006932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a93108-e4b5-41af-ae1a-7c203fd82965-maas-api-tls\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.015001 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.014973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr9s\" (UniqueName: \"kubernetes.io/projected/06a93108-e4b5-41af-ae1a-7c203fd82965-kube-api-access-7nr9s\") pod \"maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.099235 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.099162    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:51.244196 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:51.244168    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"]
Jul 24 02:15:51.245390 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:15:51.245368    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a93108_e4b5_41af_ae1a_7c203fd82965.slice/crio-4fabdb97afcfa85e81998e0195216523df2b74948d48ac973d179500d6f1a988 WatchSource:0}: Error finding container 4fabdb97afcfa85e81998e0195216523df2b74948d48ac973d179500d6f1a988: Status 404 returned error can't find the container with id 4fabdb97afcfa85e81998e0195216523df2b74948d48ac973d179500d6f1a988
Jul 24 02:15:52.253754 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:52.253710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7" event={"ID":"06a93108-e4b5-41af-ae1a-7c203fd82965","Type":"ContainerStarted","Data":"f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a"}
Jul 24 02:15:52.254174 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:52.253810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7" event={"ID":"06a93108-e4b5-41af-ae1a-7c203fd82965","Type":"ContainerStarted","Data":"4fabdb97afcfa85e81998e0195216523df2b74948d48ac973d179500d6f1a988"}
Jul 24 02:15:52.254174 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:52.253874    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:15:52.274183 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:52.274095    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7" podStartSLOduration=2.012798995 podStartE2EDuration="2.274076759s" totalImagesPullingTime="261.277764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:15:52.270961687 +0000 UTC m=+2772.693028522" watchObservedRunningTime="2026-07-24 02:15:52.274076759 +0000 UTC m=+2772.696143597"
Jul 24 02:15:58.267407 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:15:58.267377    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:16:15.963244 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:15.963169    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tmzhk_639e4494-3463-4707-9680-2a2ccc9d555c/payload-processing/0.log"
Jul 24 02:16:57.864094 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:57.864019    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tmzhk_639e4494-3463-4707-9680-2a2ccc9d555c/payload-processing/0.log"
Jul 24 02:16:58.601375 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.601337    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"]
Jul 24 02:16:58.605417 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.605392    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.607979 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.607953    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e50c2c-openshift-default-dockercfg-tl5mg\""
Jul 24 02:16:58.624590 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.624554    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"]
Jul 24 02:16:58.682150 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-envoy\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682176    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-certs\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-socket\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-token\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-data\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7t4\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-kube-api-access-mz7t4\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682667 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-credential-socket\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.682667 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.682425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-podinfo\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783634 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783593    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-credential-socket\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783834 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-podinfo\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783834 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-envoy\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-certs\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-socket\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.783952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-token\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784205 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-data\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784205 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.783997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7t4\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-kube-api-access-mz7t4\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784205 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.784077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-credential-socket\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784355 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.784294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-certs\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.784369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-socket\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784424 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.784419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-data\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.784537 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.784514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.786202 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.786182    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-envoy\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.786605 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.786582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-podinfo\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.792652 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.792633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-token\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.792791 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.792774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7t4\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-kube-api-access-mz7t4\") pod \"e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") " pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:58.916881 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:58.916793    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:16:59.079470 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.079440    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"]
Jul 24 02:16:59.080212 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:16:59.080186    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bd3a3e_0a2e_49a3_be78_5d0421101567.slice/crio-4f22774822b3d7ad388c3c4b64ce5001f5b0f01283161185ac5099360999a336 WatchSource:0}: Error finding container 4f22774822b3d7ad388c3c4b64ce5001f5b0f01283161185ac5099360999a336: Status 404 returned error can't find the container with id 4f22774822b3d7ad388c3c4b64ce5001f5b0f01283161185ac5099360999a336
Jul 24 02:16:59.082568 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.082524    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:16:59.082647 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.082623    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:16:59.082688 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.082669    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:16:59.688665 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.688626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" event={"ID":"c9bd3a3e-0a2e-49a3-be78-5d0421101567","Type":"ContainerStarted","Data":"0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"}
Jul 24 02:16:59.688869 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.688695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" event={"ID":"c9bd3a3e-0a2e-49a3-be78-5d0421101567","Type":"ContainerStarted","Data":"4f22774822b3d7ad388c3c4b64ce5001f5b0f01283161185ac5099360999a336"}
Jul 24 02:16:59.713995 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.713943    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" podStartSLOduration=1.713926459 podStartE2EDuration="1.713926459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:16:59.71093171 +0000 UTC m=+2840.132998570" watchObservedRunningTime="2026-07-24 02:16:59.713926459 +0000 UTC m=+2840.135993295"
Jul 24 02:16:59.917836 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:16:59.917801    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:00.923220 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:00.923190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:01.701585 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:01.701555    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:01.702498 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:01.702473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:04.267264 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.267226    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"]
Jul 24 02:17:04.271623 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.271600    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.274197 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.274177    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e50c2c-serving-cert\""
Jul 24 02:17:04.280787 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.280491    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"]
Jul 24 02:17:04.342354 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.342317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3050d5-2897-4a85-971f-985365de1292-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.342526 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.342454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3050d5-2897-4a85-971f-985365de1292-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.342526 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.342503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzxd\" (UniqueName: \"kubernetes.io/projected/fb3050d5-2897-4a85-971f-985365de1292-kube-api-access-nbzxd\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.443386 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.443345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3050d5-2897-4a85-971f-985365de1292-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.443596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.443403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzxd\" (UniqueName: \"kubernetes.io/projected/fb3050d5-2897-4a85-971f-985365de1292-kube-api-access-nbzxd\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.443596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.443467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3050d5-2897-4a85-971f-985365de1292-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.444148 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.444123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3050d5-2897-4a85-971f-985365de1292-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.446018 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.445994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3050d5-2897-4a85-971f-985365de1292-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.451657 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.451629    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzxd\" (UniqueName: \"kubernetes.io/projected/fb3050d5-2897-4a85-971f-985365de1292-kube-api-access-nbzxd\") pod \"maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.586062 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.585958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:04.734794 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:04.734766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"]
Jul 24 02:17:04.736569 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:17:04.736542    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3050d5_2897_4a85_971f_985365de1292.slice/crio-77b387b572e37cb05f23092d9115ab7c8f0ecca1cfbeb10f650511ba2519ca51 WatchSource:0}: Error finding container 77b387b572e37cb05f23092d9115ab7c8f0ecca1cfbeb10f650511ba2519ca51: Status 404 returned error can't find the container with id 77b387b572e37cb05f23092d9115ab7c8f0ecca1cfbeb10f650511ba2519ca51
Jul 24 02:17:05.733193 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:05.733154    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj" event={"ID":"fb3050d5-2897-4a85-971f-985365de1292","Type":"ContainerStarted","Data":"fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4"}
Jul 24 02:17:05.733606 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:05.733216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj" event={"ID":"fb3050d5-2897-4a85-971f-985365de1292","Type":"ContainerStarted","Data":"77b387b572e37cb05f23092d9115ab7c8f0ecca1cfbeb10f650511ba2519ca51"}
Jul 24 02:17:05.733606 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:05.733360    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:05.751230 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:05.751168    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj" podStartSLOduration=1.326089029 podStartE2EDuration="1.751151688s" totalImagesPullingTime="425.062659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:17:05.750375607 +0000 UTC m=+2846.172442464" watchObservedRunningTime="2026-07-24 02:17:05.751151688 +0000 UTC m=+2846.173218530"
Jul 24 02:17:11.744827 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:11.744791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:21.843770 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:21.843679    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerID="0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c" exitCode=0
Jul 24 02:17:21.844323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:21.843760    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" event={"ID":"c9bd3a3e-0a2e-49a3-be78-5d0421101567","Type":"ContainerDied","Data":"0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"}
Jul 24 02:17:21.844323 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:21.844269    2564 scope.go:117] "RemoveContainer" containerID="0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"
Jul 24 02:17:21.845629 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:21.845603    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:17:21.845700 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:21.845671    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:17:21.845737 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:21.845704    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 24 02:17:22.331268 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.331237    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"]
Jul 24 02:17:22.334141 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.334125    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:22.336495 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.336475    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:17:22.341850 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.341823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"]
Jul 24 02:17:22.430325 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.430272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4716110b-eb87-4319-9fd3-f7c163c7a8bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz\" (UID: \"4716110b-eb87-4319-9fd3-f7c163c7a8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:22.531346 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.531288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4716110b-eb87-4319-9fd3-f7c163c7a8bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz\" (UID: \"4716110b-eb87-4319-9fd3-f7c163c7a8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:22.532407 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.532373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4716110b-eb87-4319-9fd3-f7c163c7a8bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz\" (UID: \"4716110b-eb87-4319-9fd3-f7c163c7a8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:22.647782 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.647685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:22.841653 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:17:22.841599    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4716110b_eb87_4319_9fd3_f7c163c7a8bf.slice/crio-ae8c82e5c9adf45a88e91c06b1712558b2f18f621ee35dededd73f36bed7d7d9 WatchSource:0}: Error finding container ae8c82e5c9adf45a88e91c06b1712558b2f18f621ee35dededd73f36bed7d7d9: Status 404 returned error can't find the container with id ae8c82e5c9adf45a88e91c06b1712558b2f18f621ee35dededd73f36bed7d7d9
Jul 24 02:17:22.842800 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.842760    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"]
Jul 24 02:17:22.854310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.854274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" event={"ID":"c9bd3a3e-0a2e-49a3-be78-5d0421101567","Type":"ContainerStarted","Data":"49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255"}
Jul 24 02:17:22.855769 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.855740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz" event={"ID":"4716110b-eb87-4319-9fd3-f7c163c7a8bf","Type":"ContainerStarted","Data":"ae8c82e5c9adf45a88e91c06b1712558b2f18f621ee35dededd73f36bed7d7d9"}
Jul 24 02:17:22.926118 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.917493    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:22.926118 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.917623    2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Jul 24 02:17:22.926118 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:22.917679    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Jul 24 02:17:23.863602 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:23.863561    2564 generic.go:356] "Generic (PLEG): container finished" podID="4716110b-eb87-4319-9fd3-f7c163c7a8bf" containerID="39868df7650e9dc19282c5bf0b599f7b5bed1529fdb842a8efd0a25cc8d1099c" exitCode=0
Jul 24 02:17:23.864164 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:23.863641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz" event={"ID":"4716110b-eb87-4319-9fd3-f7c163c7a8bf","Type":"ContainerDied","Data":"39868df7650e9dc19282c5bf0b599f7b5bed1529fdb842a8efd0a25cc8d1099c"}
Jul 24 02:17:23.917862 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:23.917816    2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Jul 24 02:17:23.918093 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:23.917895    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Jul 24 02:17:24.920060 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:24.918509    2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Jul 24 02:17:24.920060 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:24.918584    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Jul 24 02:17:25.065052 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.064996    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:25.163187 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.163097    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4716110b-eb87-4319-9fd3-f7c163c7a8bf-maas-api-service-ca\") pod \"4716110b-eb87-4319-9fd3-f7c163c7a8bf\" (UID: \"4716110b-eb87-4319-9fd3-f7c163c7a8bf\") "
Jul 24 02:17:25.163438 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.163415    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4716110b-eb87-4319-9fd3-f7c163c7a8bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4716110b-eb87-4319-9fd3-f7c163c7a8bf" (UID: "4716110b-eb87-4319-9fd3-f7c163c7a8bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:17:25.264615 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.264560    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4716110b-eb87-4319-9fd3-f7c163c7a8bf-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:25.877379 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.877341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz" event={"ID":"4716110b-eb87-4319-9fd3-f7c163c7a8bf","Type":"ContainerDied","Data":"ae8c82e5c9adf45a88e91c06b1712558b2f18f621ee35dededd73f36bed7d7d9"}
Jul 24 02:17:25.877560 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.877399    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae8c82e5c9adf45a88e91c06b1712558b2f18f621ee35dededd73f36bed7d7d9"
Jul 24 02:17:25.877560 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.877360    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"
Jul 24 02:17:25.917500 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.917456    2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Jul 24 02:17:25.917722 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:25.917523    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Jul 24 02:17:26.921210 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:26.921169    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:26.921692 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:26.921569    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:26.922430 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:26.922409    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:35.261875 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.261766    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"]
Jul 24 02:17:35.262467 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.262056    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj" podUID="fb3050d5-2897-4a85-971f-985365de1292" containerName="maas-api" containerID="cri-o://fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4" gracePeriod=30
Jul 24 02:17:35.548221 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.548187    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:35.577094 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.571588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3050d5-2897-4a85-971f-985365de1292-openshift-service-ca\") pod \"fb3050d5-2897-4a85-971f-985365de1292\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") "
Jul 24 02:17:35.577094 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.571678    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3050d5-2897-4a85-971f-985365de1292-maas-api-tls\") pod \"fb3050d5-2897-4a85-971f-985365de1292\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") "
Jul 24 02:17:35.577094 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.571727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbzxd\" (UniqueName: \"kubernetes.io/projected/fb3050d5-2897-4a85-971f-985365de1292-kube-api-access-nbzxd\") pod \"fb3050d5-2897-4a85-971f-985365de1292\" (UID: \"fb3050d5-2897-4a85-971f-985365de1292\") "
Jul 24 02:17:35.577094 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.575213    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3050d5-2897-4a85-971f-985365de1292-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb3050d5-2897-4a85-971f-985365de1292" (UID: "fb3050d5-2897-4a85-971f-985365de1292"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:17:35.580181 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.580119    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3050d5-2897-4a85-971f-985365de1292-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb3050d5-2897-4a85-971f-985365de1292" (UID: "fb3050d5-2897-4a85-971f-985365de1292"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:17:35.582647 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.582575    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3050d5-2897-4a85-971f-985365de1292-kube-api-access-nbzxd" (OuterVolumeSpecName: "kube-api-access-nbzxd") pod "fb3050d5-2897-4a85-971f-985365de1292" (UID: "fb3050d5-2897-4a85-971f-985365de1292"). InnerVolumeSpecName "kube-api-access-nbzxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:17:35.673575 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.673538    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbzxd\" (UniqueName: \"kubernetes.io/projected/fb3050d5-2897-4a85-971f-985365de1292-kube-api-access-nbzxd\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:35.673575 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.673568    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb3050d5-2897-4a85-971f-985365de1292-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:35.673575 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.673578    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb3050d5-2897-4a85-971f-985365de1292-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:35.949174 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.949135    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb3050d5-2897-4a85-971f-985365de1292" containerID="fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4" exitCode=0
Jul 24 02:17:35.949377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.949213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj" event={"ID":"fb3050d5-2897-4a85-971f-985365de1292","Type":"ContainerDied","Data":"fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4"}
Jul 24 02:17:35.949377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.949223    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"
Jul 24 02:17:35.949377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.949306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj" event={"ID":"fb3050d5-2897-4a85-971f-985365de1292","Type":"ContainerDied","Data":"77b387b572e37cb05f23092d9115ab7c8f0ecca1cfbeb10f650511ba2519ca51"}
Jul 24 02:17:35.949377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.949338    2564 scope.go:117] "RemoveContainer" containerID="fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4"
Jul 24 02:17:35.959902 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.959877    2564 scope.go:117] "RemoveContainer" containerID="fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4"
Jul 24 02:17:35.960238 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:17:35.960215    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4\": container with ID starting with fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4 not found: ID does not exist" containerID="fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4"
Jul 24 02:17:35.960313 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.960247    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4"} err="failed to get container status \"fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4\": rpc error: code = NotFound desc = could not find container \"fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4\": container with ID starting with fea538ce5cab5ae47e5ec4495505dcf497dffd351ad896682cc43a249ee2caa4 not found: ID does not exist"
Jul 24 02:17:35.971828 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.971793    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"]
Jul 24 02:17:35.974853 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:35.974829    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e50c2c-65479d588f-h68jj"]
Jul 24 02:17:36.211102 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:36.210990    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3050d5-2897-4a85-971f-985365de1292" path="/var/lib/kubelet/pods/fb3050d5-2897-4a85-971f-985365de1292/volumes"
Jul 24 02:17:37.471401 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:37.471362    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"]
Jul 24 02:17:37.475831 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:37.475800    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e50c2c-v6lwz"]
Jul 24 02:17:38.210511 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:38.210476    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4716110b-eb87-4319-9fd3-f7c163c7a8bf" path="/var/lib/kubelet/pods/4716110b-eb87-4319-9fd3-f7c163c7a8bf/volumes"
Jul 24 02:17:44.160098 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:44.160056    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"]
Jul 24 02:17:44.160489 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:44.160335    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy" containerID="cri-o://49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255" gracePeriod=30
Jul 24 02:17:48.117801 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.117761    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"]
Jul 24 02:17:48.118263 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.118249    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4716110b-eb87-4319-9fd3-f7c163c7a8bf" containerName="revoke-keys"
Jul 24 02:17:48.118310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.118265    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4716110b-eb87-4319-9fd3-f7c163c7a8bf" containerName="revoke-keys"
Jul 24 02:17:48.118310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.118289    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3050d5-2897-4a85-971f-985365de1292" containerName="maas-api"
Jul 24 02:17:48.118310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.118295    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3050d5-2897-4a85-971f-985365de1292" containerName="maas-api"
Jul 24 02:17:48.118403 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.118369    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3050d5-2897-4a85-971f-985365de1292" containerName="maas-api"
Jul 24 02:17:48.118403 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.118381    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4716110b-eb87-4319-9fd3-f7c163c7a8bf" containerName="revoke-keys"
Jul 24 02:17:48.123160 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.123132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:48.125725 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.125697    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:17:48.129409 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.129375    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"]
Jul 24 02:17:48.199668 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.199628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05153fc-9750-47e6-9397-375ea8a8f882-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv\" (UID: \"a05153fc-9750-47e6-9397-375ea8a8f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:48.300377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.300324    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05153fc-9750-47e6-9397-375ea8a8f882-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv\" (UID: \"a05153fc-9750-47e6-9397-375ea8a8f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:48.300973 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.300948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05153fc-9750-47e6-9397-375ea8a8f882-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv\" (UID: \"a05153fc-9750-47e6-9397-375ea8a8f882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:48.439439 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.439341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:48.596891 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:48.596860    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"]
Jul 24 02:17:48.597353 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:17:48.597326    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05153fc_9750_47e6_9397_375ea8a8f882.slice/crio-da796d8b7e231d37960f6ab1c169bd73dc74ac8b24900ae08f3357999e19dc8c WatchSource:0}: Error finding container da796d8b7e231d37960f6ab1c169bd73dc74ac8b24900ae08f3357999e19dc8c: Status 404 returned error can't find the container with id da796d8b7e231d37960f6ab1c169bd73dc74ac8b24900ae08f3357999e19dc8c
Jul 24 02:17:49.049583 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.049543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv" event={"ID":"a05153fc-9750-47e6-9397-375ea8a8f882","Type":"ContainerStarted","Data":"cb6af0d7b32d171b5e8379a28ba15fbc3fbb4fb9a364925007b8e72b04590810"}
Jul 24 02:17:49.049765 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.049602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv" event={"ID":"a05153fc-9750-47e6-9397-375ea8a8f882","Type":"ContainerStarted","Data":"da796d8b7e231d37960f6ab1c169bd73dc74ac8b24900ae08f3357999e19dc8c"}
Jul 24 02:17:49.067496 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.067438    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv" podStartSLOduration=1.067418403 podStartE2EDuration="1.067418403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:17:49.066343619 +0000 UTC m=+2889.488410479" watchObservedRunningTime="2026-07-24 02:17:49.067418403 +0000 UTC m=+2889.489485274"
Jul 24 02:17:49.420778 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.420748    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:49.511439 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511400    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-certs\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511439 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511440    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-data\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511483    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istiod-ca-cert\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511521    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-podinfo\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511559    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-envoy\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511584    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-token\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511627    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-socket\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511671 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz7t4\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-kube-api-access-mz7t4\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511693    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-credential-socket\") pod \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\" (UID: \"c9bd3a3e-0a2e-49a3-be78-5d0421101567\") "
Jul 24 02:17:49.511996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511805    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:17:49.511996 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.511931    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:17:49.512215 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.512067    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:17:49.512215 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.512156    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istiod-ca-cert\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.512215 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.512178    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-certs\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.512359 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.512294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:17:49.512628 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.512601    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:17:49.513989 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.513961    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:17:49.513989 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.513968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:17:49.514257 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.514240    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 02:17:49.514335 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.514309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-kube-api-access-mz7t4" (OuterVolumeSpecName: "kube-api-access-mz7t4") pod "c9bd3a3e-0a2e-49a3-be78-5d0421101567" (UID: "c9bd3a3e-0a2e-49a3-be78-5d0421101567"). InnerVolumeSpecName "kube-api-access-mz7t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:17:49.613279 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613237    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-workload-socket\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.613279 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613270    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz7t4\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-kube-api-access-mz7t4\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.613279 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613283    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-credential-socket\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.613510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613295    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-data\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.613510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613308    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-podinfo\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.613510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613319    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-envoy\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:49.613510 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:49.613330    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bd3a3e-0a2e-49a3-be78-5d0421101567-istio-token\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:50.059566 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.059528    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerID="49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255" exitCode=0
Jul 24 02:17:50.059779 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.059591    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"
Jul 24 02:17:50.059779 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.059609    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" event={"ID":"c9bd3a3e-0a2e-49a3-be78-5d0421101567","Type":"ContainerDied","Data":"49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255"}
Jul 24 02:17:50.059779 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.059668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft" event={"ID":"c9bd3a3e-0a2e-49a3-be78-5d0421101567","Type":"ContainerDied","Data":"4f22774822b3d7ad388c3c4b64ce5001f5b0f01283161185ac5099360999a336"}
Jul 24 02:17:50.059779 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.059703    2564 scope.go:117] "RemoveContainer" containerID="49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255"
Jul 24 02:17:50.061561 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.061536    2564 generic.go:356] "Generic (PLEG): container finished" podID="a05153fc-9750-47e6-9397-375ea8a8f882" containerID="cb6af0d7b32d171b5e8379a28ba15fbc3fbb4fb9a364925007b8e72b04590810" exitCode=0
Jul 24 02:17:50.061635 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.061591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv" event={"ID":"a05153fc-9750-47e6-9397-375ea8a8f882","Type":"ContainerDied","Data":"cb6af0d7b32d171b5e8379a28ba15fbc3fbb4fb9a364925007b8e72b04590810"}
Jul 24 02:17:50.071203 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.071180    2564 scope.go:117] "RemoveContainer" containerID="0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"
Jul 24 02:17:50.081630 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.081608    2564 scope.go:117] "RemoveContainer" containerID="49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255"
Jul 24 02:17:50.081932 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:17:50.081914    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255\": container with ID starting with 49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255 not found: ID does not exist" containerID="49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255"
Jul 24 02:17:50.082037 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.081942    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255"} err="failed to get container status \"49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255\": rpc error: code = NotFound desc = could not find container \"49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255\": container with ID starting with 49680226422a19acf8612e85b58c1c4c6f03c078d3ef531e426a5a904f240255 not found: ID does not exist"
Jul 24 02:17:50.082037 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.081960    2564 scope.go:117] "RemoveContainer" containerID="0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"
Jul 24 02:17:50.082240 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:17:50.082220    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c\": container with ID starting with 0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c not found: ID does not exist" containerID="0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"
Jul 24 02:17:50.082306 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.082246    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c"} err="failed to get container status \"0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c\": rpc error: code = NotFound desc = could not find container \"0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c\": container with ID starting with 0bedaf632f99e896f436a82e5f8d547586ff39063e4aaa26b243362f22aa5d8c not found: ID does not exist"
Jul 24 02:17:50.095596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.095556    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"]
Jul 24 02:17:50.102531 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.102500    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e50c2c-openshift-default-84dfdd5b76-fvwft"]
Jul 24 02:17:50.211544 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:50.211513    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" path="/var/lib/kubelet/pods/c9bd3a3e-0a2e-49a3-be78-5d0421101567/volumes"
Jul 24 02:17:51.219495 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:51.219464    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:51.332613 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:51.332518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05153fc-9750-47e6-9397-375ea8a8f882-maas-api-service-ca\") pod \"a05153fc-9750-47e6-9397-375ea8a8f882\" (UID: \"a05153fc-9750-47e6-9397-375ea8a8f882\") "
Jul 24 02:17:51.332924 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:51.332896    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05153fc-9750-47e6-9397-375ea8a8f882-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a05153fc-9750-47e6-9397-375ea8a8f882" (UID: "a05153fc-9750-47e6-9397-375ea8a8f882"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:17:51.433977 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:51.433946    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05153fc-9750-47e6-9397-375ea8a8f882-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:17:52.082422 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:52.082380    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv" event={"ID":"a05153fc-9750-47e6-9397-375ea8a8f882","Type":"ContainerDied","Data":"da796d8b7e231d37960f6ab1c169bd73dc74ac8b24900ae08f3357999e19dc8c"}
Jul 24 02:17:52.082422 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:52.082403    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"
Jul 24 02:17:52.082644 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:17:52.082434    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da796d8b7e231d37960f6ab1c169bd73dc74ac8b24900ae08f3357999e19dc8c"
Jul 24 02:18:08.428478 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.428436    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"]
Jul 24 02:18:08.429345 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.429308    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p" podUID="63509298-50c1-43db-b47f-52137811bcb8" containerName="maas-api" containerID="cri-o://34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6" gracePeriod=30
Jul 24 02:18:08.492007 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.491962    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"]
Jul 24 02:18:08.492310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.492277    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d" podUID="324359aa-48e2-40d4-8dec-724224295a4f" containerName="payload-pre-processing" containerID="cri-o://92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6" gracePeriod=30
Jul 24 02:18:08.801974 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.801944    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:18:08.805555 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.805519    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:18:08.813995 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.813955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63509298-50c1-43db-b47f-52137811bcb8-openshift-service-ca\") pod \"63509298-50c1-43db-b47f-52137811bcb8\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") "
Jul 24 02:18:08.814189 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5nq\" (UniqueName: \"kubernetes.io/projected/63509298-50c1-43db-b47f-52137811bcb8-kube-api-access-qn5nq\") pod \"63509298-50c1-43db-b47f-52137811bcb8\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") "
Jul 24 02:18:08.814189 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814081    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63509298-50c1-43db-b47f-52137811bcb8-maas-api-tls\") pod \"63509298-50c1-43db-b47f-52137811bcb8\" (UID: \"63509298-50c1-43db-b47f-52137811bcb8\") "
Jul 24 02:18:08.814189 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814105    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324359aa-48e2-40d4-8dec-724224295a4f-plugins-config-volume\") pod \"324359aa-48e2-40d4-8dec-724224295a4f\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") "
Jul 24 02:18:08.814360 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j6cn\" (UniqueName: \"kubernetes.io/projected/324359aa-48e2-40d4-8dec-724224295a4f-kube-api-access-9j6cn\") pod \"324359aa-48e2-40d4-8dec-724224295a4f\" (UID: \"324359aa-48e2-40d4-8dec-724224295a4f\") "
Jul 24 02:18:08.814414 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814376    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63509298-50c1-43db-b47f-52137811bcb8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63509298-50c1-43db-b47f-52137811bcb8" (UID: "63509298-50c1-43db-b47f-52137811bcb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:18:08.814522 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814495    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324359aa-48e2-40d4-8dec-724224295a4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "324359aa-48e2-40d4-8dec-724224295a4f" (UID: "324359aa-48e2-40d4-8dec-724224295a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:18:08.814669 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814641    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/324359aa-48e2-40d4-8dec-724224295a4f-plugins-config-volume\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:18:08.814669 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.814664    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63509298-50c1-43db-b47f-52137811bcb8-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:18:08.816462 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.816436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63509298-50c1-43db-b47f-52137811bcb8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63509298-50c1-43db-b47f-52137811bcb8" (UID: "63509298-50c1-43db-b47f-52137811bcb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:18:08.816570 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.816489    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63509298-50c1-43db-b47f-52137811bcb8-kube-api-access-qn5nq" (OuterVolumeSpecName: "kube-api-access-qn5nq") pod "63509298-50c1-43db-b47f-52137811bcb8" (UID: "63509298-50c1-43db-b47f-52137811bcb8"). InnerVolumeSpecName "kube-api-access-qn5nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:18:08.816570 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.816500    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324359aa-48e2-40d4-8dec-724224295a4f-kube-api-access-9j6cn" (OuterVolumeSpecName: "kube-api-access-9j6cn") pod "324359aa-48e2-40d4-8dec-724224295a4f" (UID: "324359aa-48e2-40d4-8dec-724224295a4f"). InnerVolumeSpecName "kube-api-access-9j6cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:18:08.915474 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.915434    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5nq\" (UniqueName: \"kubernetes.io/projected/63509298-50c1-43db-b47f-52137811bcb8-kube-api-access-qn5nq\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:18:08.915474 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.915468    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63509298-50c1-43db-b47f-52137811bcb8-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:18:08.915474 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:08.915478    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j6cn\" (UniqueName: \"kubernetes.io/projected/324359aa-48e2-40d4-8dec-724224295a4f-kube-api-access-9j6cn\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:18:09.194518 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.194482    2564 generic.go:356] "Generic (PLEG): container finished" podID="63509298-50c1-43db-b47f-52137811bcb8" containerID="34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6" exitCode=0
Jul 24 02:18:09.194726 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.194537    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"
Jul 24 02:18:09.194726 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.194572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p" event={"ID":"63509298-50c1-43db-b47f-52137811bcb8","Type":"ContainerDied","Data":"34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6"}
Jul 24 02:18:09.194726 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.194634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p" event={"ID":"63509298-50c1-43db-b47f-52137811bcb8","Type":"ContainerDied","Data":"324dc7e7e132dd71313688fb39d0f007158ed217216139da105f94da61aea90d"}
Jul 24 02:18:09.194726 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.194665    2564 scope.go:117] "RemoveContainer" containerID="34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6"
Jul 24 02:18:09.196077 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.196011    2564 generic.go:356] "Generic (PLEG): container finished" podID="324359aa-48e2-40d4-8dec-724224295a4f" containerID="92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6" exitCode=0
Jul 24 02:18:09.196077 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.196069    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d" event={"ID":"324359aa-48e2-40d4-8dec-724224295a4f","Type":"ContainerDied","Data":"92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6"}
Jul 24 02:18:09.196242 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.196076    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"
Jul 24 02:18:09.196242 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.196121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d" event={"ID":"324359aa-48e2-40d4-8dec-724224295a4f","Type":"ContainerDied","Data":"3d7fb6a92cd7e87a0845431a8a66408f44c175eac2966820ad8f9e8cc93cd427"}
Jul 24 02:18:09.206247 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.206225    2564 scope.go:117] "RemoveContainer" containerID="34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6"
Jul 24 02:18:09.206585 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:18:09.206564    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6\": container with ID starting with 34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6 not found: ID does not exist" containerID="34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6"
Jul 24 02:18:09.206636 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.206595    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6"} err="failed to get container status \"34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6\": rpc error: code = NotFound desc = could not find container \"34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6\": container with ID starting with 34a59ada42021d40a8782181b394af43185217184bb665d7102c5a834842e4a6 not found: ID does not exist"
Jul 24 02:18:09.206636 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.206614    2564 scope.go:117] "RemoveContainer" containerID="92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6"
Jul 24 02:18:09.219571 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.219546    2564 scope.go:117] "RemoveContainer" containerID="92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6"
Jul 24 02:18:09.219966 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:18:09.219940    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6\": container with ID starting with 92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6 not found: ID does not exist" containerID="92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6"
Jul 24 02:18:09.220090 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.219976    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6"} err="failed to get container status \"92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6\": rpc error: code = NotFound desc = could not find container \"92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6\": container with ID starting with 92c69e7b91fb7d3fb68e9cedbbdf108f9b57820ce4e71fec0c022d490e65dab6 not found: ID does not exist"
Jul 24 02:18:09.221866 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.221825    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"]
Jul 24 02:18:09.226218 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.226186    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a002f7-688c94b584-l5c4d"]
Jul 24 02:18:09.238011 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.237974    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"]
Jul 24 02:18:09.241810 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:09.241782    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a002f7-5648486d76-47h9p"]
Jul 24 02:18:10.211241 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:10.211206    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324359aa-48e2-40d4-8dec-724224295a4f" path="/var/lib/kubelet/pods/324359aa-48e2-40d4-8dec-724224295a4f/volumes"
Jul 24 02:18:10.211720 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:10.211635    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63509298-50c1-43db-b47f-52137811bcb8" path="/var/lib/kubelet/pods/63509298-50c1-43db-b47f-52137811bcb8/volumes"
Jul 24 02:18:13.230007 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:13.229968    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"]
Jul 24 02:18:13.232791 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:13.232767    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a002f7-t4xvv"]
Jul 24 02:18:14.210536 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:18:14.210497    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05153fc-9750-47e6-9397-375ea8a8f882" path="/var/lib/kubelet/pods/a05153fc-9750-47e6-9397-375ea8a8f882/volumes"
Jul 24 02:19:02.744610 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.744524    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"]
Jul 24 02:19:02.745004 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.744979    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63509298-50c1-43db-b47f-52137811bcb8" containerName="maas-api"
Jul 24 02:19:02.745004 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.744992    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="63509298-50c1-43db-b47f-52137811bcb8" containerName="maas-api"
Jul 24 02:19:02.745004 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745003    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324359aa-48e2-40d4-8dec-724224295a4f" containerName="payload-pre-processing"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745008    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="324359aa-48e2-40d4-8dec-724224295a4f" containerName="payload-pre-processing"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745021    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745042    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745048    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05153fc-9750-47e6-9397-375ea8a8f882" containerName="revoke-keys"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745053    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05153fc-9750-47e6-9397-375ea8a8f882" containerName="revoke-keys"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745071    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy"
Jul 24 02:19:02.745135 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745076    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy"
Jul 24 02:19:02.745346 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745144    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="324359aa-48e2-40d4-8dec-724224295a4f" containerName="payload-pre-processing"
Jul 24 02:19:02.745346 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745152    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy"
Jul 24 02:19:02.745346 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745157    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05153fc-9750-47e6-9397-375ea8a8f882" containerName="revoke-keys"
Jul 24 02:19:02.745346 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.745164    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="63509298-50c1-43db-b47f-52137811bcb8" containerName="maas-api"
Jul 24 02:19:02.748327 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.748308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:02.750657 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.750629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:19:02.757180 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.757151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"]
Jul 24 02:19:02.865651 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.865610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e08db-f6cb-4552-8745-5a04d2327909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg\" (UID: \"552e08db-f6cb-4552-8745-5a04d2327909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:02.966437 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.966400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e08db-f6cb-4552-8745-5a04d2327909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg\" (UID: \"552e08db-f6cb-4552-8745-5a04d2327909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:02.967059 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:02.967018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e08db-f6cb-4552-8745-5a04d2327909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg\" (UID: \"552e08db-f6cb-4552-8745-5a04d2327909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:03.061308 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:03.061227    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:03.206806 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:03.206780    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"]
Jul 24 02:19:03.209078 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:19:03.209047    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552e08db_f6cb_4552_8745_5a04d2327909.slice/crio-76ea1ec4d4c8af542c356d6303178a8b8bb468a05affe8840cac2ebed484dedd WatchSource:0}: Error finding container 76ea1ec4d4c8af542c356d6303178a8b8bb468a05affe8840cac2ebed484dedd: Status 404 returned error can't find the container with id 76ea1ec4d4c8af542c356d6303178a8b8bb468a05affe8840cac2ebed484dedd
Jul 24 02:19:03.551698 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:03.551662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg" event={"ID":"552e08db-f6cb-4552-8745-5a04d2327909","Type":"ContainerStarted","Data":"c96904beea86fefe7d890cda59220a0331a54bc651f6452327c34605bed689a4"}
Jul 24 02:19:03.551934 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:03.551712    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg" event={"ID":"552e08db-f6cb-4552-8745-5a04d2327909","Type":"ContainerStarted","Data":"76ea1ec4d4c8af542c356d6303178a8b8bb468a05affe8840cac2ebed484dedd"}
Jul 24 02:19:03.570404 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:03.570335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg" podStartSLOduration=1.5703172749999998 podStartE2EDuration="1.570317275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:19:03.568018833 +0000 UTC m=+2963.990085669" watchObservedRunningTime="2026-07-24 02:19:03.570317275 +0000 UTC m=+2963.992384111"
Jul 24 02:19:04.559413 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:04.559377    2564 generic.go:356] "Generic (PLEG): container finished" podID="552e08db-f6cb-4552-8745-5a04d2327909" containerID="c96904beea86fefe7d890cda59220a0331a54bc651f6452327c34605bed689a4" exitCode=0
Jul 24 02:19:04.559809 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:04.559447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg" event={"ID":"552e08db-f6cb-4552-8745-5a04d2327909","Type":"ContainerDied","Data":"c96904beea86fefe7d890cda59220a0331a54bc651f6452327c34605bed689a4"}
Jul 24 02:19:05.711356 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:05.711328    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:05.795694 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:05.795656    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e08db-f6cb-4552-8745-5a04d2327909-maas-api-service-ca\") pod \"552e08db-f6cb-4552-8745-5a04d2327909\" (UID: \"552e08db-f6cb-4552-8745-5a04d2327909\") "
Jul 24 02:19:05.796056 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:05.795996    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552e08db-f6cb-4552-8745-5a04d2327909-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "552e08db-f6cb-4552-8745-5a04d2327909" (UID: "552e08db-f6cb-4552-8745-5a04d2327909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:05.896756 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:05.896667    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/552e08db-f6cb-4552-8745-5a04d2327909-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:06.574052 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:06.573936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg" event={"ID":"552e08db-f6cb-4552-8745-5a04d2327909","Type":"ContainerDied","Data":"76ea1ec4d4c8af542c356d6303178a8b8bb468a05affe8840cac2ebed484dedd"}
Jul 24 02:19:06.574052 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:06.573993    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ea1ec4d4c8af542c356d6303178a8b8bb468a05affe8840cac2ebed484dedd"
Jul 24 02:19:06.574052 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:06.573952    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"
Jul 24 02:19:13.760307 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:13.760270    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"]
Jul 24 02:19:13.760907 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:13.760779    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7" podUID="06a93108-e4b5-41af-ae1a-7c203fd82965" containerName="maas-api" containerID="cri-o://f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a" gracePeriod=30
Jul 24 02:19:14.028647 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.028577    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:19:14.182297 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.182251    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a93108-e4b5-41af-ae1a-7c203fd82965-maas-api-tls\") pod \"06a93108-e4b5-41af-ae1a-7c203fd82965\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") "
Jul 24 02:19:14.182486 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.182392    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nr9s\" (UniqueName: \"kubernetes.io/projected/06a93108-e4b5-41af-ae1a-7c203fd82965-kube-api-access-7nr9s\") pod \"06a93108-e4b5-41af-ae1a-7c203fd82965\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") "
Jul 24 02:19:14.182558 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.182485    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a93108-e4b5-41af-ae1a-7c203fd82965-openshift-service-ca\") pod \"06a93108-e4b5-41af-ae1a-7c203fd82965\" (UID: \"06a93108-e4b5-41af-ae1a-7c203fd82965\") "
Jul 24 02:19:14.182846 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.182813    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a93108-e4b5-41af-ae1a-7c203fd82965-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06a93108-e4b5-41af-ae1a-7c203fd82965" (UID: "06a93108-e4b5-41af-ae1a-7c203fd82965"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:14.185000 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.184972    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a93108-e4b5-41af-ae1a-7c203fd82965-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06a93108-e4b5-41af-ae1a-7c203fd82965" (UID: "06a93108-e4b5-41af-ae1a-7c203fd82965"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:19:14.185000 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.184984    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a93108-e4b5-41af-ae1a-7c203fd82965-kube-api-access-7nr9s" (OuterVolumeSpecName: "kube-api-access-7nr9s") pod "06a93108-e4b5-41af-ae1a-7c203fd82965" (UID: "06a93108-e4b5-41af-ae1a-7c203fd82965"). InnerVolumeSpecName "kube-api-access-7nr9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:19:14.284420 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.284326    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a93108-e4b5-41af-ae1a-7c203fd82965-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:14.284420 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.284357    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a93108-e4b5-41af-ae1a-7c203fd82965-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:14.284420 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.284366    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nr9s\" (UniqueName: \"kubernetes.io/projected/06a93108-e4b5-41af-ae1a-7c203fd82965-kube-api-access-7nr9s\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:14.627797 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.627703    2564 generic.go:356] "Generic (PLEG): container finished" podID="06a93108-e4b5-41af-ae1a-7c203fd82965" containerID="f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a" exitCode=0
Jul 24 02:19:14.627797 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.627754    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"
Jul 24 02:19:14.627797 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.627770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7" event={"ID":"06a93108-e4b5-41af-ae1a-7c203fd82965","Type":"ContainerDied","Data":"f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a"}
Jul 24 02:19:14.628073 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.627824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7" event={"ID":"06a93108-e4b5-41af-ae1a-7c203fd82965","Type":"ContainerDied","Data":"4fabdb97afcfa85e81998e0195216523df2b74948d48ac973d179500d6f1a988"}
Jul 24 02:19:14.628073 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.627855    2564 scope.go:117] "RemoveContainer" containerID="f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a"
Jul 24 02:19:14.637793 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.637774    2564 scope.go:117] "RemoveContainer" containerID="f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a"
Jul 24 02:19:14.638196 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:19:14.638175    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a\": container with ID starting with f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a not found: ID does not exist" containerID="f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a"
Jul 24 02:19:14.638257 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.638205    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a"} err="failed to get container status \"f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a\": rpc error: code = NotFound desc = could not find container \"f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a\": container with ID starting with f4a0ea0f2e83be551e1dad23fb2f826a5ec68c3c6eed2acf5238a3b41331341a not found: ID does not exist"
Jul 24 02:19:14.646717 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.646688    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"]
Jul 24 02:19:14.650660 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:14.650639    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-477d74-57bd57df4-l55k7"]
Jul 24 02:19:16.210264 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:16.210226    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06a93108-e4b5-41af-ae1a-7c203fd82965" path="/var/lib/kubelet/pods/06a93108-e4b5-41af-ae1a-7c203fd82965/volumes"
Jul 24 02:19:17.893606 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:17.893564    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"]
Jul 24 02:19:17.896222 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:17.896199    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-477d74-lzpkg"]
Jul 24 02:19:18.210597 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:18.210564    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552e08db-f6cb-4552-8745-5a04d2327909" path="/var/lib/kubelet/pods/552e08db-f6cb-4552-8745-5a04d2327909/volumes"
Jul 24 02:19:24.850402 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850367    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"]
Jul 24 02:19:24.850818 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06a93108-e4b5-41af-ae1a-7c203fd82965" containerName="maas-api"
Jul 24 02:19:24.850818 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850820    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="06a93108-e4b5-41af-ae1a-7c203fd82965" containerName="maas-api"
Jul 24 02:19:24.850895 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850832    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552e08db-f6cb-4552-8745-5a04d2327909" containerName="revoke-keys"
Jul 24 02:19:24.850895 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850838    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="552e08db-f6cb-4552-8745-5a04d2327909" containerName="revoke-keys"
Jul 24 02:19:24.850962 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850929    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9bd3a3e-0a2e-49a3-be78-5d0421101567" containerName="istio-proxy"
Jul 24 02:19:24.850962 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850939    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="552e08db-f6cb-4552-8745-5a04d2327909" containerName="revoke-keys"
Jul 24 02:19:24.850962 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.850946    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="06a93108-e4b5-41af-ae1a-7c203fd82965" containerName="maas-api"
Jul 24 02:19:24.854156 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.854126    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:24.856447 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.856423    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:19:24.861633 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.861608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"]
Jul 24 02:19:24.899907 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:24.899878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde6ea74-2b5a-4d3c-8603-bb3b434bda45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs\" (UID: \"fde6ea74-2b5a-4d3c-8603-bb3b434bda45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:25.001312 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.001272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde6ea74-2b5a-4d3c-8603-bb3b434bda45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs\" (UID: \"fde6ea74-2b5a-4d3c-8603-bb3b434bda45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:25.001899 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.001877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde6ea74-2b5a-4d3c-8603-bb3b434bda45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs\" (UID: \"fde6ea74-2b5a-4d3c-8603-bb3b434bda45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:25.169045 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.168947    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:25.310324 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.310299    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"]
Jul 24 02:19:25.312131 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:19:25.312083    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde6ea74_2b5a_4d3c_8603_bb3b434bda45.slice/crio-f4d418be453d8d1018efd81d512fe0c56d44df21c1a25de95566747b0548be09 WatchSource:0}: Error finding container f4d418be453d8d1018efd81d512fe0c56d44df21c1a25de95566747b0548be09: Status 404 returned error can't find the container with id f4d418be453d8d1018efd81d512fe0c56d44df21c1a25de95566747b0548be09
Jul 24 02:19:25.699985 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.699934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs" event={"ID":"fde6ea74-2b5a-4d3c-8603-bb3b434bda45","Type":"ContainerStarted","Data":"60a34c53ca36e8030359a57b600f41140b4ffd5c8d12996fa0455274ce354f80"}
Jul 24 02:19:25.700162 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.699995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs" event={"ID":"fde6ea74-2b5a-4d3c-8603-bb3b434bda45","Type":"ContainerStarted","Data":"f4d418be453d8d1018efd81d512fe0c56d44df21c1a25de95566747b0548be09"}
Jul 24 02:19:25.716242 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:25.716192    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs" podStartSLOduration=1.7161759490000001 podStartE2EDuration="1.716175949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 02:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:19:25.714414753 +0000 UTC m=+2986.136481605" watchObservedRunningTime="2026-07-24 02:19:25.716175949 +0000 UTC m=+2986.138242826"
Jul 24 02:19:26.708105 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:26.708066    2564 generic.go:356] "Generic (PLEG): container finished" podID="fde6ea74-2b5a-4d3c-8603-bb3b434bda45" containerID="60a34c53ca36e8030359a57b600f41140b4ffd5c8d12996fa0455274ce354f80" exitCode=0
Jul 24 02:19:26.708507 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:26.708147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs" event={"ID":"fde6ea74-2b5a-4d3c-8603-bb3b434bda45","Type":"ContainerDied","Data":"60a34c53ca36e8030359a57b600f41140b4ffd5c8d12996fa0455274ce354f80"}
Jul 24 02:19:27.857241 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:27.857215    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:27.930979 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:27.930940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde6ea74-2b5a-4d3c-8603-bb3b434bda45-maas-api-service-ca\") pod \"fde6ea74-2b5a-4d3c-8603-bb3b434bda45\" (UID: \"fde6ea74-2b5a-4d3c-8603-bb3b434bda45\") "
Jul 24 02:19:27.931337 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:27.931315    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde6ea74-2b5a-4d3c-8603-bb3b434bda45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fde6ea74-2b5a-4d3c-8603-bb3b434bda45" (UID: "fde6ea74-2b5a-4d3c-8603-bb3b434bda45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:28.032585 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:28.032500    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde6ea74-2b5a-4d3c-8603-bb3b434bda45-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:28.721113 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:28.721081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs" event={"ID":"fde6ea74-2b5a-4d3c-8603-bb3b434bda45","Type":"ContainerDied","Data":"f4d418be453d8d1018efd81d512fe0c56d44df21c1a25de95566747b0548be09"}
Jul 24 02:19:28.721291 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:28.721116    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"
Jul 24 02:19:28.721291 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:28.721154    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d418be453d8d1018efd81d512fe0c56d44df21c1a25de95566747b0548be09"
Jul 24 02:19:35.044105 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.044064    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"]
Jul 24 02:19:35.044606 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.044347    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" podUID="a2aa36a1-5350-4d93-9721-6c24562a36e5" containerName="maas-api" containerID="cri-o://638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0" gracePeriod=30
Jul 24 02:19:35.073940 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.073911    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"]
Jul 24 02:19:35.079158 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.079124    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6bcabd-29747655-hb8ll"]
Jul 24 02:19:35.306976 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.306879    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:19:35.406902 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.406861    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2mr2\" (UniqueName: \"kubernetes.io/projected/a2aa36a1-5350-4d93-9721-6c24562a36e5-kube-api-access-w2mr2\") pod \"a2aa36a1-5350-4d93-9721-6c24562a36e5\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") "
Jul 24 02:19:35.407117 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.406928    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls\") pod \"a2aa36a1-5350-4d93-9721-6c24562a36e5\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") "
Jul 24 02:19:35.407117 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.406979    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa36a1-5350-4d93-9721-6c24562a36e5-openshift-service-ca\") pod \"a2aa36a1-5350-4d93-9721-6c24562a36e5\" (UID: \"a2aa36a1-5350-4d93-9721-6c24562a36e5\") "
Jul 24 02:19:35.407409 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.407381    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2aa36a1-5350-4d93-9721-6c24562a36e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2aa36a1-5350-4d93-9721-6c24562a36e5" (UID: "a2aa36a1-5350-4d93-9721-6c24562a36e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:35.409330 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.409303    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aa36a1-5350-4d93-9721-6c24562a36e5-kube-api-access-w2mr2" (OuterVolumeSpecName: "kube-api-access-w2mr2") pod "a2aa36a1-5350-4d93-9721-6c24562a36e5" (UID: "a2aa36a1-5350-4d93-9721-6c24562a36e5"). InnerVolumeSpecName "kube-api-access-w2mr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:19:35.409442 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.409302    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2aa36a1-5350-4d93-9721-6c24562a36e5" (UID: "a2aa36a1-5350-4d93-9721-6c24562a36e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:19:35.507928 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.507887    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2mr2\" (UniqueName: \"kubernetes.io/projected/a2aa36a1-5350-4d93-9721-6c24562a36e5-kube-api-access-w2mr2\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:35.507928 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.507921    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa36a1-5350-4d93-9721-6c24562a36e5-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:35.507928 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.507935    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa36a1-5350-4d93-9721-6c24562a36e5-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:35.767750 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.767713    2564 generic.go:356] "Generic (PLEG): container finished" podID="a2aa36a1-5350-4d93-9721-6c24562a36e5" containerID="638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0" exitCode=0
Jul 24 02:19:35.767958 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.767774    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"
Jul 24 02:19:35.767958 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.767799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" event={"ID":"a2aa36a1-5350-4d93-9721-6c24562a36e5","Type":"ContainerDied","Data":"638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0"}
Jul 24 02:19:35.767958 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.767895    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj" event={"ID":"a2aa36a1-5350-4d93-9721-6c24562a36e5","Type":"ContainerDied","Data":"a4002d4cc6ceaf0fc3bcc74d2f7b3b6cf0c49e3bea09fda540c1f2c84285b7a2"}
Jul 24 02:19:35.767958 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.767930    2564 scope.go:117] "RemoveContainer" containerID="638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0"
Jul 24 02:19:35.778846 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.778828    2564 scope.go:117] "RemoveContainer" containerID="638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0"
Jul 24 02:19:35.779176 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:19:35.779140    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0\": container with ID starting with 638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0 not found: ID does not exist" containerID="638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0"
Jul 24 02:19:35.779240 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.779188    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0"} err="failed to get container status \"638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0\": rpc error: code = NotFound desc = could not find container \"638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0\": container with ID starting with 638b61c932f0038a09f6c3c9806d4cccefe643f99af6e49059fd2d54e1bbd0b0 not found: ID does not exist"
Jul 24 02:19:35.789876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.789849    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"]
Jul 24 02:19:35.793511 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:35.793487    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6bcabd-55d6fbb99b-ddpqj"]
Jul 24 02:19:36.210434 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:36.210402    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2aa36a1-5350-4d93-9721-6c24562a36e5" path="/var/lib/kubelet/pods/a2aa36a1-5350-4d93-9721-6c24562a36e5/volumes"
Jul 24 02:19:36.210814 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:36.210711    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8587860-e3b4-4d55-bafe-9c7978758b71" path="/var/lib/kubelet/pods/b8587860-e3b4-4d55-bafe-9c7978758b71/volumes"
Jul 24 02:19:39.945894 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:39.945855    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"]
Jul 24 02:19:39.948516 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:39.948496    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6bcabd-bdsxs"]
Jul 24 02:19:40.212183 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:40.212104    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fde6ea74-2b5a-4d3c-8603-bb3b434bda45" path="/var/lib/kubelet/pods/fde6ea74-2b5a-4d3c-8603-bb3b434bda45/volumes"
Jul 24 02:19:40.670264 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:40.670186    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:19:40.677005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:40.676984    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:19:40.696214 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:40.696191    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:19:40.703061 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:40.703040    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85shc_31b8df2f-e374-4426-b777-905a07afac34/ovn-acl-logging/0.log"
Jul 24 02:19:46.844910 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:46.844873    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"]
Jul 24 02:19:46.845556 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:46.845087    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" containerID="cri-o://208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741" gracePeriod=30
Jul 24 02:19:47.243080 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243038    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"]
Jul 24 02:19:47.243580 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243560    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2aa36a1-5350-4d93-9721-6c24562a36e5" containerName="maas-api"
Jul 24 02:19:47.243677 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243582    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2aa36a1-5350-4d93-9721-6c24562a36e5" containerName="maas-api"
Jul 24 02:19:47.243677 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243603    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde6ea74-2b5a-4d3c-8603-bb3b434bda45" containerName="revoke-keys"
Jul 24 02:19:47.243677 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243611    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde6ea74-2b5a-4d3c-8603-bb3b434bda45" containerName="revoke-keys"
Jul 24 02:19:47.243843 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243723    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2aa36a1-5350-4d93-9721-6c24562a36e5" containerName="maas-api"
Jul 24 02:19:47.243843 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.243738    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde6ea74-2b5a-4d3c-8603-bb3b434bda45" containerName="revoke-keys"
Jul 24 02:19:47.248942 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.248921    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:47.251529 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.251505    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gcl7q\""
Jul 24 02:19:47.254159 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.254121    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"]
Jul 24 02:19:47.424186 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.424153    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6a41c7-0cc4-4173-a70f-42a2cb17a525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx\" (UID: \"6e6a41c7-0cc4-4173-a70f-42a2cb17a525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:47.525515 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.525407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6a41c7-0cc4-4173-a70f-42a2cb17a525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx\" (UID: \"6e6a41c7-0cc4-4173-a70f-42a2cb17a525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:47.526077 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.526057    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6a41c7-0cc4-4173-a70f-42a2cb17a525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx\" (UID: \"6e6a41c7-0cc4-4173-a70f-42a2cb17a525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:47.562816 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.562781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:47.707220 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.707189    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"]
Jul 24 02:19:47.708903 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:19:47.708874    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6a41c7_0cc4_4173_a70f_42a2cb17a525.slice/crio-6e8f7d4b064afb0cb645caf09eaa3383812123c3f79876f799dfc646071dcf47 WatchSource:0}: Error finding container 6e8f7d4b064afb0cb645caf09eaa3383812123c3f79876f799dfc646071dcf47: Status 404 returned error can't find the container with id 6e8f7d4b064afb0cb645caf09eaa3383812123c3f79876f799dfc646071dcf47
Jul 24 02:19:47.846590 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:47.846497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx" event={"ID":"6e6a41c7-0cc4-4173-a70f-42a2cb17a525","Type":"ContainerStarted","Data":"6e8f7d4b064afb0cb645caf09eaa3383812123c3f79876f799dfc646071dcf47"}
Jul 24 02:19:48.379059 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:48.379004    2564 patch_prober.go:28] interesting pod/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 02:19:48.379243 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:48.379079    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 02:19:48.854021 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:48.853982    2564 generic.go:356] "Generic (PLEG): container finished" podID="6e6a41c7-0cc4-4173-a70f-42a2cb17a525" containerID="e1b89c38d2a55017cbcda4a1f640ff4277db42f7cce5798d14375d1bcb8ded31" exitCode=0
Jul 24 02:19:48.854416 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:48.854062    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx" event={"ID":"6e6a41c7-0cc4-4173-a70f-42a2cb17a525","Type":"ContainerDied","Data":"e1b89c38d2a55017cbcda4a1f640ff4277db42f7cce5798d14375d1bcb8ded31"}
Jul 24 02:19:50.009437 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.009408    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:50.151463 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.151372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6a41c7-0cc4-4173-a70f-42a2cb17a525-maas-api-service-ca\") pod \"6e6a41c7-0cc4-4173-a70f-42a2cb17a525\" (UID: \"6e6a41c7-0cc4-4173-a70f-42a2cb17a525\") "
Jul 24 02:19:50.151727 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.151705    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6a41c7-0cc4-4173-a70f-42a2cb17a525-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e6a41c7-0cc4-4173-a70f-42a2cb17a525" (UID: "6e6a41c7-0cc4-4173-a70f-42a2cb17a525"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:50.252736 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.252708    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6a41c7-0cc4-4173-a70f-42a2cb17a525-maas-api-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:50.868725 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.868695    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"
Jul 24 02:19:50.868916 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.868735    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx" event={"ID":"6e6a41c7-0cc4-4173-a70f-42a2cb17a525","Type":"ContainerDied","Data":"6e8f7d4b064afb0cb645caf09eaa3383812123c3f79876f799dfc646071dcf47"}
Jul 24 02:19:50.868916 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:50.868792    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8f7d4b064afb0cb645caf09eaa3383812123c3f79876f799dfc646071dcf47"
Jul 24 02:19:52.116992 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.116924    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:19:52.268805 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.268765    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-certs\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.268805 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.268808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-socket\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269165 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.268876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-data\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269165 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.268906    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-envoy\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269165 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.268974    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf2dk\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-kube-api-access-xf2dk\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269165 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269048    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-credential-socket\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269165 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269105    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:19:52.269429 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-token\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269429 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istiod-ca-cert\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269429 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-podinfo\") pod \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\" (UID: \"f0329af3-a7fe-48bd-a856-c287c6c11fd5\") "
Jul 24 02:19:52.269693 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269672    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-certs\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.269754 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269291    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:19:52.269754 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.269577    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:52.270203 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.270159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:19:52.270519 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.270491    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:19:52.271679 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.271654    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 02:19:52.271932 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.271761    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:19:52.271932 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.271786    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-kube-api-access-xf2dk" (OuterVolumeSpecName: "kube-api-access-xf2dk") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "kube-api-access-xf2dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:19:52.272106 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.271959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0329af3-a7fe-48bd-a856-c287c6c11fd5" (UID: "f0329af3-a7fe-48bd-a856-c287c6c11fd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 02:19:52.370370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370281    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-workload-socket\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370319    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-data\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370333    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-envoy\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370347    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2dk\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-kube-api-access-xf2dk\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370360    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0329af3-a7fe-48bd-a856-c287c6c11fd5-credential-socket\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370370 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370373    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-token\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370673 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370388    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istiod-ca-cert\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.370673 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.370400    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0329af3-a7fe-48bd-a856-c287c6c11fd5-istio-podinfo\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:52.882971 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.882930    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerID="208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741" exitCode=0
Jul 24 02:19:52.883193 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.882996    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"
Jul 24 02:19:52.883193 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.883019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" event={"ID":"f0329af3-a7fe-48bd-a856-c287c6c11fd5","Type":"ContainerDied","Data":"208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741"}
Jul 24 02:19:52.883193 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.883137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv" event={"ID":"f0329af3-a7fe-48bd-a856-c287c6c11fd5","Type":"ContainerDied","Data":"8e6986e055d8a668e5350eaf84aa989b25cf77fc03c2871f6b6fa5927855d023"}
Jul 24 02:19:52.883193 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.883168    2564 scope.go:117] "RemoveContainer" containerID="208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741"
Jul 24 02:19:52.894298 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.894282    2564 scope.go:117] "RemoveContainer" containerID="802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"
Jul 24 02:19:52.903933 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.903908    2564 scope.go:117] "RemoveContainer" containerID="208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741"
Jul 24 02:19:52.904259 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:19:52.904234    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741\": container with ID starting with 208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741 not found: ID does not exist" containerID="208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741"
Jul 24 02:19:52.904334 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.904271    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741"} err="failed to get container status \"208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741\": rpc error: code = NotFound desc = could not find container \"208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741\": container with ID starting with 208653776aef8f93fe795c7633e21421bec072dde5e35c674d42b0b0905d7741 not found: ID does not exist"
Jul 24 02:19:52.904334 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.904296    2564 scope.go:117] "RemoveContainer" containerID="802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"
Jul 24 02:19:52.904553 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:19:52.904535    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f\": container with ID starting with 802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f not found: ID does not exist" containerID="802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"
Jul 24 02:19:52.904607 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.904559    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f"} err="failed to get container status \"802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f\": rpc error: code = NotFound desc = could not find container \"802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f\": container with ID starting with 802187f91d21c83486a58e2f1d8d2e0f66fbebb35be0120779046d087d3a0b9f not found: ID does not exist"
Jul 24 02:19:52.905877 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.905853    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"]
Jul 24 02:19:52.908529 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:52.908510    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6bcabd-openshift-default-6f8dff4848-s5ttv"]
Jul 24 02:19:54.211242 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:54.211198    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" path="/var/lib/kubelet/pods/f0329af3-a7fe-48bd-a856-c287c6c11fd5/volumes"
Jul 24 02:19:57.599426 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.599391    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"]
Jul 24 02:19:57.599924 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.599748    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" podUID="825cabe5-6b08-492e-ba96-497335fb0cb8" containerName="maas-api" containerID="cri-o://17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971" gracePeriod=30
Jul 24 02:19:57.620110 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.620082    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"]
Jul 24 02:19:57.626547 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.626523    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a7fbc1-29747655-wj9bc"]
Jul 24 02:19:57.880358 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.880332    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:19:57.927287 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.927240    2564 generic.go:356] "Generic (PLEG): container finished" podID="825cabe5-6b08-492e-ba96-497335fb0cb8" containerID="17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971" exitCode=0
Jul 24 02:19:57.927287 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.927287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" event={"ID":"825cabe5-6b08-492e-ba96-497335fb0cb8","Type":"ContainerDied","Data":"17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971"}
Jul 24 02:19:57.927538 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.927300    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"
Jul 24 02:19:57.927538 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.927340    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2" event={"ID":"825cabe5-6b08-492e-ba96-497335fb0cb8","Type":"ContainerDied","Data":"04283bd3d59cb1697478abcb3d5f01bf5580fd42a4559a98f18d28a03b6c94d9"}
Jul 24 02:19:57.927538 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.927374    2564 scope.go:117] "RemoveContainer" containerID="17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971"
Jul 24 02:19:57.938835 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.938816    2564 scope.go:117] "RemoveContainer" containerID="17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971"
Jul 24 02:19:57.939168 ip-10-0-129-127 kubenswrapper[2564]: E0724 02:19:57.939147    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971\": container with ID starting with 17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971 not found: ID does not exist" containerID="17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971"
Jul 24 02:19:57.939216 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:57.939177    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971"} err="failed to get container status \"17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971\": rpc error: code = NotFound desc = could not find container \"17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971\": container with ID starting with 17bedefe8b148243f233d482f898e6bef2879dbefc34a6737c9b1d3ccf50c971 not found: ID does not exist"
Jul 24 02:19:58.023718 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.023686    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825cabe5-6b08-492e-ba96-497335fb0cb8-openshift-service-ca\") pod \"825cabe5-6b08-492e-ba96-497335fb0cb8\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") "
Jul 24 02:19:58.023876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.023739    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") pod \"825cabe5-6b08-492e-ba96-497335fb0cb8\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") "
Jul 24 02:19:58.023876 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.023775    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl8qz\" (UniqueName: \"kubernetes.io/projected/825cabe5-6b08-492e-ba96-497335fb0cb8-kube-api-access-dl8qz\") pod \"825cabe5-6b08-492e-ba96-497335fb0cb8\" (UID: \"825cabe5-6b08-492e-ba96-497335fb0cb8\") "
Jul 24 02:19:58.024151 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.024125    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825cabe5-6b08-492e-ba96-497335fb0cb8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "825cabe5-6b08-492e-ba96-497335fb0cb8" (UID: "825cabe5-6b08-492e-ba96-497335fb0cb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 02:19:58.025985 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.025959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "825cabe5-6b08-492e-ba96-497335fb0cb8" (UID: "825cabe5-6b08-492e-ba96-497335fb0cb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 02:19:58.026088 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.026011    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825cabe5-6b08-492e-ba96-497335fb0cb8-kube-api-access-dl8qz" (OuterVolumeSpecName: "kube-api-access-dl8qz") pod "825cabe5-6b08-492e-ba96-497335fb0cb8" (UID: "825cabe5-6b08-492e-ba96-497335fb0cb8"). InnerVolumeSpecName "kube-api-access-dl8qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 02:19:58.125311 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.125220    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825cabe5-6b08-492e-ba96-497335fb0cb8-openshift-service-ca\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:58.125311 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.125248    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825cabe5-6b08-492e-ba96-497335fb0cb8-maas-api-tls\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:58.125311 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.125271    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8qz\" (UniqueName: \"kubernetes.io/projected/825cabe5-6b08-492e-ba96-497335fb0cb8-kube-api-access-dl8qz\") on node \"ip-10-0-129-127.ec2.internal\" DevicePath \"\""
Jul 24 02:19:58.210464 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.210425    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a4fe33-d9f4-4d15-a7c2-124937df54d7" path="/var/lib/kubelet/pods/e0a4fe33-d9f4-4d15-a7c2-124937df54d7/volumes"
Jul 24 02:19:58.247830 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.247794    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"]
Jul 24 02:19:58.250939 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:19:58.250906    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a7fbc1-769db465df-lhwb2"]
Jul 24 02:20:00.210812 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:00.210765    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825cabe5-6b08-492e-ba96-497335fb0cb8" path="/var/lib/kubelet/pods/825cabe5-6b08-492e-ba96-497335fb0cb8/volumes"
Jul 24 02:20:02.323151 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:02.323111    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"]
Jul 24 02:20:02.325543 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:02.325522    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a7fbc1-nm9nx"]
Jul 24 02:20:04.210281 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:04.210251    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e6a41c7-0cc4-4173-a70f-42a2cb17a525" path="/var/lib/kubelet/pods/6e6a41c7-0cc4-4173-a70f-42a2cb17a525/volumes"
Jul 24 02:20:15.332155 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:15.332126    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tdqpq_5205735c-72f0-4f80-9938-665014506e2f/manager/0.log"
Jul 24 02:20:15.461532 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:15.461500    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d9cdf4547-tglhm_55c23306-95d0-40e3-99eb-82bb0b50ff48/manager/0.log"
Jul 24 02:20:15.578752 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:15.578726    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ddw4l_0d964262-f632-47c5-8c90-aa3be7424715/server/0.log"
Jul 24 02:20:15.707728 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:15.707703    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sn9r6_779f50f6-88e4-4731-ad1d-70901f32e0a9/manager/0.log"
Jul 24 02:20:16.096933 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.096862    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59c5678768-lxb4h_3b89af98-48a7-4539-8b9c-222585ec8154/copy-manifests/0.log"
Jul 24 02:20:16.103618 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.103599    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59c5678768-lxb4h_3b89af98-48a7-4539-8b9c-222585ec8154/manager/0.log"
Jul 24 02:20:16.459704 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.459676    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67d795fcc8-sc4gd_a80ff20c-12e1-4bd1-a14d-f50c734e0756/maas-api/0.log"
Jul 24 02:20:16.580326 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.580298    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747625-bt8lq_317a1758-c518-4615-a147-b46ae78073ce/cleanup/0.log"
Jul 24 02:20:16.692870 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.692839    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747640-rfk4p_5bf6eb04-f811-4ebe-847c-d508e6114da0/cleanup/0.log"
Jul 24 02:20:16.806847 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.806766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747655-srjh5_e350cac9-965b-4d77-9964-32af3e21675b/cleanup/0.log"
Jul 24 02:20:16.927340 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:16.927310    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4pr2x_35029611-0208-4b7c-9057-0a068d54e3ce/postgres/0.log"
Jul 24 02:20:17.728365 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:17.728335    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw_cc00c075-0552-47df-8578-a5996bde6f76/util/0.log"
Jul 24 02:20:17.735469 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:17.735443    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw_cc00c075-0552-47df-8578-a5996bde6f76/pull/0.log"
Jul 24 02:20:17.743184 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:17.743162    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw_cc00c075-0552-47df-8578-a5996bde6f76/extract/0.log"
Jul 24 02:20:17.871117 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:17.871086    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882_4dfc01fd-8856-4a6d-ac2a-141b0c2545f5/util/0.log"
Jul 24 02:20:17.877768 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:17.877745    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882_4dfc01fd-8856-4a6d-ac2a-141b0c2545f5/pull/0.log"
Jul 24 02:20:17.884292 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:17.884272    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882_4dfc01fd-8856-4a6d-ac2a-141b0c2545f5/extract/0.log"
Jul 24 02:20:18.009920 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.009845    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s_fd8be8f8-4127-4b32-8a63-12fa11d6065d/util/0.log"
Jul 24 02:20:18.016073 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.016047    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s_fd8be8f8-4127-4b32-8a63-12fa11d6065d/pull/0.log"
Jul 24 02:20:18.021915 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.021892    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s_fd8be8f8-4127-4b32-8a63-12fa11d6065d/extract/0.log"
Jul 24 02:20:18.141016 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.140985    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw_cf33f2fe-f4d5-4d39-9d6c-78307c1d284f/util/0.log"
Jul 24 02:20:18.147734 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.147697    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw_cf33f2fe-f4d5-4d39-9d6c-78307c1d284f/pull/0.log"
Jul 24 02:20:18.153803 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.153786    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw_cf33f2fe-f4d5-4d39-9d6c-78307c1d284f/extract/0.log"
Jul 24 02:20:18.398095 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:18.398002    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5v684_58bbc007-999c-4148-a2c5-13768ec27358/manager/0.log"
Jul 24 02:20:19.641033 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:19.640983    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-msljx_b22c323c-8c8c-48df-9563-0ab7ea6b3868/discovery/0.log"
Jul 24 02:20:19.752202 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:19.752170    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8544c7785d-7xqm5_58b537b9-80f4-48c8-8d79-32605a755b58/kube-auth-proxy/0.log"
Jul 24 02:20:19.987508 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:19.987476    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/1.log"
Jul 24 02:20:20.231567 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:20.231532    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tmzhk_639e4494-3463-4707-9680-2a2ccc9d555c/payload-processing/0.log"
Jul 24 02:20:20.703722 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:20.703688    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mgw8r_19c18e68-73fe-4683-8910-9d0126fb5bcf/main/0.log"
Jul 24 02:20:29.748182 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:29.748150    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2dzp8_ef9426fc-2fc6-41c7-a11b-51e7f2c1fb53/global-pull-secret-syncer/0.log"
Jul 24 02:20:29.953582 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:29.953550    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dhr7p_eeead595-ea5f-4a2e-a731-bb8a56c7702a/konnectivity-agent/0.log"
Jul 24 02:20:29.973231 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:29.973199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-127.ec2.internal_496a950db9b46b88536b00684a81f3f6/haproxy/0.log"
Jul 24 02:20:33.935521 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:33.935418    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw_cc00c075-0552-47df-8578-a5996bde6f76/extract/0.log"
Jul 24 02:20:33.960115 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:33.960084    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw_cc00c075-0552-47df-8578-a5996bde6f76/util/0.log"
Jul 24 02:20:33.980013 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:33.979987    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592tjfw_cc00c075-0552-47df-8578-a5996bde6f76/pull/0.log"
Jul 24 02:20:34.013011 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.012984    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882_4dfc01fd-8856-4a6d-ac2a-141b0c2545f5/extract/0.log"
Jul 24 02:20:34.035547 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.035516    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882_4dfc01fd-8856-4a6d-ac2a-141b0c2545f5/util/0.log"
Jul 24 02:20:34.058714 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.058694    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08c882_4dfc01fd-8856-4a6d-ac2a-141b0c2545f5/pull/0.log"
Jul 24 02:20:34.089489 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.089469    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s_fd8be8f8-4127-4b32-8a63-12fa11d6065d/extract/0.log"
Jul 24 02:20:34.110220 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.110194    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s_fd8be8f8-4127-4b32-8a63-12fa11d6065d/util/0.log"
Jul 24 02:20:34.132522 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.132498    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735gr2s_fd8be8f8-4127-4b32-8a63-12fa11d6065d/pull/0.log"
Jul 24 02:20:34.164653 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.164627    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw_cf33f2fe-f4d5-4d39-9d6c-78307c1d284f/extract/0.log"
Jul 24 02:20:34.186906 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.186820    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw_cf33f2fe-f4d5-4d39-9d6c-78307c1d284f/util/0.log"
Jul 24 02:20:34.217567 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.217532    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16wgjw_cf33f2fe-f4d5-4d39-9d6c-78307c1d284f/pull/0.log"
Jul 24 02:20:34.575632 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:34.575608    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5v684_58bbc007-999c-4148-a2c5-13768ec27358/manager/0.log"
Jul 24 02:20:36.948168 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:36.948139    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xbqbq_d9085d8f-b04c-4c20-80e8-c8201f767063/kube-state-metrics/0.log"
Jul 24 02:20:36.971755 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:36.971731    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xbqbq_d9085d8f-b04c-4c20-80e8-c8201f767063/kube-rbac-proxy-main/0.log"
Jul 24 02:20:36.996962 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:36.996937    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xbqbq_d9085d8f-b04c-4c20-80e8-c8201f767063/kube-rbac-proxy-self/0.log"
Jul 24 02:20:37.032807 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.032782    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f86786855-m5886_d326a642-01d8-4477-bc8c-d6d918ff8130/metrics-server/0.log"
Jul 24 02:20:37.190822 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.190741    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sh4bl_35a3a0b8-4f5d-4580-b7bc-e6c241104637/node-exporter/0.log"
Jul 24 02:20:37.214470 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.214440    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sh4bl_35a3a0b8-4f5d-4580-b7bc-e6c241104637/kube-rbac-proxy/0.log"
Jul 24 02:20:37.245919 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.245895    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sh4bl_35a3a0b8-4f5d-4580-b7bc-e6c241104637/init-textfile/0.log"
Jul 24 02:20:37.721888 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.721854    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-sgwxx_f444e058-9325-428d-9061-bee7ea74df28/prometheus-operator-admission-webhook/0.log"
Jul 24 02:20:37.754092 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.754060    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f444cc7f-wb5nt_87ad5f7b-e7aa-4b26-95e5-f4d1959e8484/telemeter-client/0.log"
Jul 24 02:20:37.777557 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.777523    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f444cc7f-wb5nt_87ad5f7b-e7aa-4b26-95e5-f4d1959e8484/reload/0.log"
Jul 24 02:20:37.800746 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.800713    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75f444cc7f-wb5nt_87ad5f7b-e7aa-4b26-95e5-f4d1959e8484/kube-rbac-proxy/0.log"
Jul 24 02:20:37.832774 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.832744    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-964f86565-phch7_ba3ca594-7a95-4965-a2c9-ae5b9e09794d/thanos-query/0.log"
Jul 24 02:20:37.854073 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.854048    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-964f86565-phch7_ba3ca594-7a95-4965-a2c9-ae5b9e09794d/kube-rbac-proxy-web/0.log"
Jul 24 02:20:37.873005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.872982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-964f86565-phch7_ba3ca594-7a95-4965-a2c9-ae5b9e09794d/kube-rbac-proxy/0.log"
Jul 24 02:20:37.893747 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.893726    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-964f86565-phch7_ba3ca594-7a95-4965-a2c9-ae5b9e09794d/prom-label-proxy/0.log"
Jul 24 02:20:37.915325 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.915298    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-964f86565-phch7_ba3ca594-7a95-4965-a2c9-ae5b9e09794d/kube-rbac-proxy-rules/0.log"
Jul 24 02:20:37.935244 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:37.935218    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-964f86565-phch7_ba3ca594-7a95-4965-a2c9-ae5b9e09794d/kube-rbac-proxy-metrics/0.log"
Jul 24 02:20:39.224636 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:39.224604    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-rcnft_3e0f32d1-116e-47b7-be49-9928e551ca9e/networking-console-plugin/0.log"
Jul 24 02:20:39.848862 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:39.848833    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/1.log"
Jul 24 02:20:39.854751 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:39.854726    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ppgdx_f9f43754-34d2-4c03-8535-5b4b5208db5c/console-operator/2.log"
Jul 24 02:20:40.492377 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.492346    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"]
Jul 24 02:20:40.492970 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.492949    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.492973    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.492987    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825cabe5-6b08-492e-ba96-497335fb0cb8" containerName="maas-api"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.492996    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="825cabe5-6b08-492e-ba96-497335fb0cb8" containerName="maas-api"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493007    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493015    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493041    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e6a41c7-0cc4-4173-a70f-42a2cb17a525" containerName="revoke-keys"
Jul 24 02:20:40.493085 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493051    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e6a41c7-0cc4-4173-a70f-42a2cb17a525" containerName="revoke-keys"
Jul 24 02:20:40.493450 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493160    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e6a41c7-0cc4-4173-a70f-42a2cb17a525" containerName="revoke-keys"
Jul 24 02:20:40.493450 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493172    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="825cabe5-6b08-492e-ba96-497335fb0cb8" containerName="maas-api"
Jul 24 02:20:40.493450 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493183    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy"
Jul 24 02:20:40.493450 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.493193    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0329af3-a7fe-48bd-a856-c287c6c11fd5" containerName="istio-proxy"
Jul 24 02:20:40.496710 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.496688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.499469 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.499425    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m522w\"/\"openshift-service-ca.crt\""
Jul 24 02:20:40.499607 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.499490    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m522w\"/\"kube-root-ca.crt\""
Jul 24 02:20:40.500384 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.500357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m522w\"/\"default-dockercfg-5qkq6\""
Jul 24 02:20:40.505445 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.505418    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"]
Jul 24 02:20:40.531937 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.531898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-lib-modules\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.532169 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.531943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-sys\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.532169 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.532071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-podres\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.532169 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.532107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-proc\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.532169 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.532144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtrv\" (UniqueName: \"kubernetes.io/projected/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-kube-api-access-qgtrv\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.632747 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-lib-modules\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.632747 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632750    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-sys\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-podres\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-proc\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtrv\" (UniqueName: \"kubernetes.io/projected/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-kube-api-access-qgtrv\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-lib-modules\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-proc\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-sys\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.633005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.632943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-podres\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.643584 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.643553    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtrv\" (UniqueName: \"kubernetes.io/projected/c89186f3-6ffe-4bd3-b760-8c381fd7f1c6-kube-api-access-qgtrv\") pod \"perf-node-gather-daemonset-62lr9\" (UID: \"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6\") " pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.807808 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.807717    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:40.952434 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:40.952398    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"]
Jul 24 02:20:40.953666 ip-10-0-129-127 kubenswrapper[2564]: W0724 02:20:40.953637    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc89186f3_6ffe_4bd3_b760_8c381fd7f1c6.slice/crio-f300482a76663d6855638f68c2d5f744216c7e05ef07bb58f157db0b9be4dc98 WatchSource:0}: Error finding container f300482a76663d6855638f68c2d5f744216c7e05ef07bb58f157db0b9be4dc98: Status 404 returned error can't find the container with id f300482a76663d6855638f68c2d5f744216c7e05ef07bb58f157db0b9be4dc98
Jul 24 02:20:41.173904 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:41.173820    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-czjnh_1325aa34-e514-4dc1-9f59-11baa8109142/volume-data-source-validator/0.log"
Jul 24 02:20:41.200636 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:41.200601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9" event={"ID":"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6","Type":"ContainerStarted","Data":"f27dc88f2b72318cac2479b9c3515de8a671ba55105a932437f963fe8e709533"}
Jul 24 02:20:41.200817 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:41.200656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9" event={"ID":"c89186f3-6ffe-4bd3-b760-8c381fd7f1c6","Type":"ContainerStarted","Data":"f300482a76663d6855638f68c2d5f744216c7e05ef07bb58f157db0b9be4dc98"}
Jul 24 02:20:41.200817 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:41.200720    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:41.216952 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:41.216905    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9" podStartSLOduration=1.216889808 podStartE2EDuration="1.216889808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 02:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 02:20:41.215923455 +0000 UTC m=+3061.637990303" watchObservedRunningTime="2026-07-24 02:20:41.216889808 +0000 UTC m=+3061.638956643"
Jul 24 02:20:42.220448 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:42.220420    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z6ckz_432593c2-05d4-4dfc-8246-87cc5b5d91c5/dns/0.log"
Jul 24 02:20:42.242404 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:42.242379    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z6ckz_432593c2-05d4-4dfc-8246-87cc5b5d91c5/kube-rbac-proxy/0.log"
Jul 24 02:20:42.333533 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:42.333503    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r8dkq_25e37d82-6241-46f1-974c-655e53ab2189/dns-node-resolver/0.log"
Jul 24 02:20:42.903551 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:42.903469    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4dtpj_c132a9e1-eb6e-4602-bfa2-40af3a44a8d6/node-ca/0.log"
Jul 24 02:20:44.023798 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:44.023762    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-msljx_b22c323c-8c8c-48df-9563-0ab7ea6b3868/discovery/0.log"
Jul 24 02:20:44.047350 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:44.047322    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8544c7785d-7xqm5_58b537b9-80f4-48c8-8d79-32605a755b58/kube-auth-proxy/0.log"
Jul 24 02:20:44.097522 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:44.097493    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/0.log"
Jul 24 02:20:44.123310 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:44.123285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/1.log"
Jul 24 02:20:44.184703 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:44.184676    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tmzhk_639e4494-3463-4707-9680-2a2ccc9d555c/payload-processing/0.log"
Jul 24 02:20:45.030054 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:45.029995    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qj8mc_797c9bee-eab6-41f0-a52a-e64cd1034f1f/serve-healthcheck-canary/0.log"
Jul 24 02:20:45.498514 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:45.498478    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-4rr9p_1f35de79-075f-4d52-b979-c0d51c69722d/insights-operator/0.log"
Jul 24 02:20:45.499754 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:45.499729    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-4rr9p_1f35de79-075f-4d52-b979-c0d51c69722d/insights-operator/1.log"
Jul 24 02:20:45.523735 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:45.523707    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2dpdk_cb4d617d-edca-4b97-a24a-23449fe67b82/kube-rbac-proxy/0.log"
Jul 24 02:20:45.544516 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:45.544491    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2dpdk_cb4d617d-edca-4b97-a24a-23449fe67b82/exporter/0.log"
Jul 24 02:20:45.567928 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:45.567904    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2dpdk_cb4d617d-edca-4b97-a24a-23449fe67b82/extractor/0.log"
Jul 24 02:20:47.217073 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:47.217047    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m522w/perf-node-gather-daemonset-62lr9"
Jul 24 02:20:48.053983 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:48.053944    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tdqpq_5205735c-72f0-4f80-9938-665014506e2f/manager/0.log"
Jul 24 02:20:48.252465 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:48.252431    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d9cdf4547-tglhm_55c23306-95d0-40e3-99eb-82bb0b50ff48/manager/0.log"
Jul 24 02:20:48.272398 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:48.272367    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ddw4l_0d964262-f632-47c5-8c90-aa3be7424715/server/0.log"
Jul 24 02:20:48.363596 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:48.363499    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sn9r6_779f50f6-88e4-4731-ad1d-70901f32e0a9/manager/0.log"
Jul 24 02:20:48.566305 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:48.566266    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59c5678768-lxb4h_3b89af98-48a7-4539-8b9c-222585ec8154/manager/0.log"
Jul 24 02:20:48.585356 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:48.585329    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59c5678768-lxb4h_3b89af98-48a7-4539-8b9c-222585ec8154/copy-manifests/0.log"
Jul 24 02:20:50.797619 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:50.797582    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-msljx_b22c323c-8c8c-48df-9563-0ab7ea6b3868/discovery/0.log"
Jul 24 02:20:50.807485 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:50.807454    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8544c7785d-7xqm5_58b537b9-80f4-48c8-8d79-32605a755b58/kube-auth-proxy/0.log"
Jul 24 02:20:50.844523 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:50.844442    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/0.log"
Jul 24 02:20:50.871362 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:50.871333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/1.log"
Jul 24 02:20:50.913333 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:50.913297    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tmzhk_639e4494-3463-4707-9680-2a2ccc9d555c/payload-processing/0.log"
Jul 24 02:20:52.436910 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:52.436772    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-msljx_b22c323c-8c8c-48df-9563-0ab7ea6b3868/discovery/0.log"
Jul 24 02:20:52.450119 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:52.450087    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8544c7785d-7xqm5_58b537b9-80f4-48c8-8d79-32605a755b58/kube-auth-proxy/0.log"
Jul 24 02:20:52.483967 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:52.483930    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/0.log"
Jul 24 02:20:52.512327 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:52.512287    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-g7hqg_3fc67a0c-d029-4f2b-afdc-72fb44fc7c17/istio-proxy/1.log"
Jul 24 02:20:52.553648 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:52.553596    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tmzhk_639e4494-3463-4707-9680-2a2ccc9d555c/payload-processing/0.log"
Jul 24 02:20:55.532152 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:55.532120    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-lq465_7a755877-b146-4230-8b54-d66daa557182/kube-storage-version-migrator-operator/1.log"
Jul 24 02:20:55.533865 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:55.533842    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-lq465_7a755877-b146-4230-8b54-d66daa557182/kube-storage-version-migrator-operator/0.log"
Jul 24 02:20:56.806933 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.806852    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/kube-multus-additional-cni-plugins/0.log"
Jul 24 02:20:56.827278 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.827243    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/egress-router-binary-copy/0.log"
Jul 24 02:20:56.848065 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.848014    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/cni-plugins/0.log"
Jul 24 02:20:56.867615 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.867584    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/bond-cni-plugin/0.log"
Jul 24 02:20:56.887054 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.887007    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/routeoverride-cni/0.log"
Jul 24 02:20:56.906803 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.906779    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/whereabouts-cni-bincopy/0.log"
Jul 24 02:20:56.928005 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:56.927972    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tj678_b7f0c07f-1466-4cdb-9f6c-bdeb36e089f5/whereabouts-cni/0.log"
Jul 24 02:20:57.031723 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:57.031688    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vlpg6_320d92f3-2f1e-45ec-8a37-fd0b31c5508f/kube-multus/0.log"
Jul 24 02:20:57.054359 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:57.054333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-crts2_c3d94894-5060-45d1-bd7b-1fea2cedeb43/network-metrics-daemon/0.log"
Jul 24 02:20:57.071821 ip-10-0-129-127 kubenswrapper[2564]: I0724 02:20:57.071740    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-crts2_c3d94894-5060-45d1-bd7b-1fea2cedeb43/kube-rbac-proxy/0.log"