Sep 10 22:59:43.547622 ip-10-0-129-219 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 22:59:43.547632 ip-10-0-129-219 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 22:59:43.547639 ip-10-0-129-219 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 22:59:43.547851 ip-10-0-129-219 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 22:59:53.797032 ip-10-0-129-219 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 22:59:53.797049 ip-10-0-129-219 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1a54b62e706d43c0b45cd7c3e02ba4be --
Sep 10 23:01:34.252455 ip-10-0-129-219 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 23:01:34.647574 ip-10-0-129-219 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:34.647574 ip-10-0-129-219 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 23:01:34.647574 ip-10-0-129-219 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:34.647574 ip-10-0-129-219 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 23:01:34.647574 ip-10-0-129-219 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:34.650032 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.649958    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 23:01:34.653778 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653764    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:34.653778 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653778    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653783    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653786    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653789    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653794    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653798    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653802    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653805    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653808    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653811    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653820    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653823    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653826    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653829    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653832    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653835    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653838    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653840    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:34.653836 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653844    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653847    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653850    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653853    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653856    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653859    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653861    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653864    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653867    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653870    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653923    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653927    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653930    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653933    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653936    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653939    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653941    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653944    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653946    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653956    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:34.654317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653958    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653961    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653965    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653968    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653971    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653974    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653976    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653979    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653982    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653985    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653988    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653990    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653993    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.653996    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654000    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654004    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654007    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654009    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654012    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:34.654801 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654015    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654017    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654020    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654023    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654026    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654028    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654031    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654033    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654036    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654038    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654041    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654044    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654046    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654050    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654053    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654057    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654059    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654062    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654064    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654067    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:34.655283 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654070    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654073    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654076    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654079    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654081    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654084    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654087    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654090    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654093    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654096    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654098    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654101    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654104    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654106    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654109    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654112    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654114    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654117    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654119    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654122    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:34.655752 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654124    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654824    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654831    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654833    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654836    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654839    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654845    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654849    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654853    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654856    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654859    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654861    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654864    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654867    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654870    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654873    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654875    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654878    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654881    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:34.656244 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654884    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654887    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654890    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654893    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654895    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654898    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654900    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654903    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654905    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654908    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654910    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654913    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654915    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654918    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654920    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654923    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654925    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654928    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654930    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654933    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:34.656743 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654943    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654946    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654948    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654951    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654953    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654956    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654959    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654961    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654964    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654967    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654971    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654974    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654977    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654979    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654982    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654984    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654987    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654990    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654993    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:34.657219 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654996    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.654998    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655002    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655005    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655008    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655011    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655013    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655016    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655018    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655021    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655024    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655026    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655031    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655034    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655042    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655045    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655047    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655050    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655053    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:34.657728 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655055    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655058    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655061    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655063    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655066    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655069    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655072    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655074    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655077    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655080    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655082    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655085    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655087    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655090    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655092    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655095    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655097    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655100    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655102    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655105    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:34.658249 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655107    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655109    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655112    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655495    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655500    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655503    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655506    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655509    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655511    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655514    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655517    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655519    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655522    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655525    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655527    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655530    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655533    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655536    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655538    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655541    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:34.658744 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655544    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655546    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655549    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655551    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655554    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655557    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655559    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655562    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655564    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655567    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655570    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655572    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655575    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655578    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655580    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655585    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655588    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655590    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655593    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655595    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:34.659236 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655598    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655602    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655606    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655609    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655611    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655614    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655617    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655619    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655622    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655625    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655627    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655631    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655635    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655638    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655641    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655644    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655647    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655650    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655653    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:34.659722 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655655    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655658    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655660    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655663    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655665    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655668    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655670    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655673    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655677    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655679    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655682    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655684    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655687    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655689    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655692    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655695    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655699    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655702    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655705    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655708    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:34.660177 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655710    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655713    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655716    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655718    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655721    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655723    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655726    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655728    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655731    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655733    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655736    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655738    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655741    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655743    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655746    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655748    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655751    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655753    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655756    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655759    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:34.660664 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655763    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655765    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.655768    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655835    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655866    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655872    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655876    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655880    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655884    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655888    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655892    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655895    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655897    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655901    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655904    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655908    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655910    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655913    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655917    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655920    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655923    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655927    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655929    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655932    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 23:01:34.661140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655936    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655939    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655943    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655945    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655948    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655952    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655955    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655957    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655961    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655964    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655967    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655971    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655974    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655977    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655979    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655982    2564 flags.go:64] FLAG: --enable-server="true"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655986    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655990    2564 flags.go:64] FLAG: --event-burst="100"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655993    2564 flags.go:64] FLAG: --event-qps="50"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.655998    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656001    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656004    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656007    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656010    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656013    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 23:01:34.661715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656016    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656019    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656022    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656025    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656028    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656031    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656034    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656037    2564 flags.go:64] FLAG: --feature-gates=""
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656041    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656044    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656046    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656049    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656052    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656056    2564 flags.go:64] FLAG: --help="false"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656059    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-219.ec2.internal"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656062    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656065    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656068    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656071    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656075    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656077    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656080    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656083    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 23:01:34.662364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656086    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656089    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656092    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656096    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656099    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656102    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656105    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656108    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656111    2564 flags.go:64] FLAG: --lock-file=""
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656114    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656117    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656120    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656125    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656128    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656130    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656133    2564 flags.go:64] FLAG: --logging-format="text"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656136    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656139    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656142    2564 flags.go:64] FLAG: --manifest-url=""
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656145    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656149    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656152    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656157    2564 flags.go:64] FLAG: --max-pods="110"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656160    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656163    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 23:01:34.662943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656166    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656169    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656172    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656175    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656178    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656184    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656187    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656204    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656209    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656214    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656219    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656224    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656227    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656230    2564 flags.go:64] FLAG: --port="10250"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656233    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656236    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-090c2f639dda37068"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656239    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656242    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656245    2564 flags.go:64] FLAG: --register-node="true"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656248    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656252    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656255    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656258    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656260    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656264    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 23:01:34.663666 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656267    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656270    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656273    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656276    2564 flags.go:64] FLAG: --runonce="false"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656279    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656282    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656286    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656289    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656292    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656295    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656298    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656302    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656305    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656308    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656311    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656314    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656317    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656320    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656324    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656329    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656332    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656335    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656339    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656342    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656345    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 23:01:34.664287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656348    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656351    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656354    2564 flags.go:64] FLAG: --v="2"
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656358    2564 flags.go:64] FLAG: --version="false"
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656362    2564 flags.go:64] FLAG: --vmodule=""
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656366    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.656369    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656473    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656476    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656480    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656483    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656487    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656491    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656494    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656497    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656500    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656502    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656505    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656508    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656510    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656513    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:34.664883 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656515    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656519    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656521    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656524    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656529    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656532    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656534    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656537    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656540    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656543    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656546    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656548    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656551    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656553    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656556    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656559    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656561    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656564    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656566    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656569    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:34.665395 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656572    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656574    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656577    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656580    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656584    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656588    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656592    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656595    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656598    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656601    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656604    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656607    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656610    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656612    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656615    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656618    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656622    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656625    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656628    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:34.666138 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656631    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656634    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656645    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656648    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656651    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656654    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656656    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656660    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656663    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656667    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656670    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656673    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656676    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656679    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656682    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656685    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656688    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656692    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656694    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:34.666733 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656697    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656700    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656703    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656706    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656708    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656711    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656714    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656716    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656719    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656721    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656725    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656728    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656737    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656740    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656743    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656746    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656749    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656751    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656754    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656757    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:34.667252 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656759    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656762    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656765    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656768    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656770    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656773    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.656775    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.657438    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.664685    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.664699    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.664715    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.664720    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664772    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664776    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664779    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:34.667777 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664782    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664785    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664788    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664791    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664794    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664796    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664799    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664802    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664805    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664808    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664811    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664813    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664816    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664819    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664821    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664824    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664826    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664829    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664832    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664834    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:34.668155 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664837    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664840    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664842    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664845    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664847    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664850    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664853    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664858    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664861    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664865    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664868    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664871    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664874    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664876    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664879    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664882    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664885    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664888    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664891    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:34.668655 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664894    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664897    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664900    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664903    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664906    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664908    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664911    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664914    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664916    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664919    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664922    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664924    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664927    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664930    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664932    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664935    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664937    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664940    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664942    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664945    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:34.669110 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664948    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664950    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664953    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664955    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664958    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664961    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664963    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664966    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664968    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664971    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664973    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664976    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664979    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664981    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664984    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664986    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664989    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664992    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664994    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.664997    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:34.669638 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665000    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665002    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665005    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665007    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665010    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665012    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665015    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665017    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665020    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665022    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665025    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665028    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665030    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665034    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665038    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665042    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:34.670115 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665046    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.665051    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665155    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665159    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665163    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665165    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665168    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665170    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665173    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665176    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665178    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665180    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665183    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665186    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665205    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:34.670525 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665209    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665211    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665214    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665217    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665219    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665222    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665224    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665227    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665230    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665232    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665235    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665237    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665240    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665243    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665247    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665254    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665257    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665259    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665262    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665265    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:34.670894 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665267    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665270    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665273    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665275    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665279    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665283    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665286    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665289    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665292    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665295    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665298    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665301    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665304    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665307    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665310    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665313    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665316    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665318    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665321    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665324    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:34.671511 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665326    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665329    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665331    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665334    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665336    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665339    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665342    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665344    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665347    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665350    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665352    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665355    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665357    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665360    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665362    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665365    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665368    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665371    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665374    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:34.671976 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665376    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665380    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665383    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665386    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665389    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665392    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665395    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665397    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665400    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665403    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665406    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665408    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665411    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665414    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665416    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665419    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665421    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665424    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665426    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:34.672452 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665429    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665431    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665434    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665437    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665439    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665442    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665444    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:34.665447    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.665452    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.667247    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.669481    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.670388    2564 server.go:1019] "Starting client certificate rotation"
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.670481    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:01:34.672896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.670520    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:01:34.673678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.673665    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:01:34.677978 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.677958    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:01:34.684675 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.684660    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 10 23:01:34.687840 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.687826    2564 log.go:25] "Validated CRI v1 image API"
Sep 10 23:01:34.688635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.688620    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 23:01:34.691290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.691273    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96a95ba7-76b3-4357-ac53-587682123203:/dev/nvme0n1p3 9b31f85b-787c-4712-a555-2d5113c2cff6:/dev/nvme0n1p4]
Sep 10 23:01:34.691352 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.691289    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 23:01:34.697827 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.697696    2564 manager.go:216] Machine: {Timestamp:2026-09-10 23:01:34.696569984 +0000 UTC m=+0.328102905 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119117 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b89ac6a9aa39366db49ba5f6e324 SystemUUID:ec25b89a-c6a9-aa39-366d-b49ba5f6e324 BootID:1a54b62e-706d-43c0-b45c-d7c3e02ba4be Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:ce:0a:8d:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:ce:0a:8d:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:8d:39:1d:fd:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 23:01:34.697827 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.697807    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 23:01:34.697990 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.697896    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:01:34.697990 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.697897    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 23:01:34.701203 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.701173    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 23:01:34.701372 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.701212    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 23:01:34.702163 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.702151    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 23:01:34.702243 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.702170    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 23:01:34.702243 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.702187    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:01:34.702935 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.702923    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:01:34.704250 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.704239    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 23:01:34.707877 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.707864    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:01:34.708018 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708007    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 23:01:34.708091 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708051    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 23:01:34.708091 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708069    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 23:01:34.708181 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708110    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 23:01:34.708181 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708123    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 10 23:01:34.708181 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708135    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 23:01:34.708653 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708639    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:01:34.708724 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.708662    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:01:34.710729 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.710710    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 23:01:34.713921 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.713905    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 23:01:34.713984 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.713937    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 23:01:34.715712 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715700    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 23:01:34.715759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715717    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 23:01:34.715759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715724    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 23:01:34.715759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715730    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 23:01:34.715759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715739    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 23:01:34.715759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715744    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 23:01:34.715759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715762    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 23:01:34.715929 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715768    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 23:01:34.715929 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715774    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 23:01:34.715929 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715780    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 23:01:34.715929 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715788    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 23:01:34.715929 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.715793    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 23:01:34.717360 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.717350    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 23:01:34.717360 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.717361    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 23:01:34.719539 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.719514    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 23:01:34.719539 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.719515    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 23:01:34.720793 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.720780    2564 server.go:1268] "Started kubelet"
Sep 10 23:01:34.720890 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.720855    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 23:01:34.720941 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.720913    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 23:01:34.720941 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.720879    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 23:01:34.721504 ip-10-0-129-219 systemd[1]: Started Kubernetes Kubelet.
Sep 10 23:01:34.724108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.723456    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 23:01:34.724108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.723740    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 23:01:34.724905 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.724888    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 23:01:34.726148 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.726131    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9bpcj"
Sep 10 23:01:34.727949 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.727933    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 23:01:34.728895 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.727989    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-219.ec2.internal.18d417f56620676f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-219.ec2.internal,UID:ip-10-0-129-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-219.ec2.internal,},FirstTimestamp:2026-09-10 23:01:34.720755567 +0000 UTC m=+0.352288489,LastTimestamp:2026-09-10 23:01:34.720755567 +0000 UTC m=+0.352288489,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-219.ec2.internal,}"
Sep 10 23:01:34.729066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729054    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 23:01:34.729118 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729072    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 23:01:34.729234 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.729146    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:34.729756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729736    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 23:01:34.729756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729742    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 23:01:34.729848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729762    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 23:01:34.729902 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729891    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 23:01:34.729945 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.729903    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 23:01:34.730580 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730565    2564 factory.go:55] Registering systemd factory
Sep 10 23:01:34.730662 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730607    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 10 23:01:34.730823 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730807    2564 factory.go:153] Registering CRI-O factory
Sep 10 23:01:34.730823 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730823    2564 factory.go:223] Registration of the crio container factory successfully
Sep 10 23:01:34.730951 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730885    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 23:01:34.730951 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730904    2564 factory.go:103] Registering Raw factory
Sep 10 23:01:34.730951 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.730920    2564 manager.go:1196] Started watching for new ooms in manager
Sep 10 23:01:34.731478 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.731464    2564 manager.go:349] Starting recovery of all containers
Sep 10 23:01:34.733675 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.733648    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 23:01:34.733798 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.733778    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 23:01:34.734129 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.734110    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9bpcj"
Sep 10 23:01:34.742688 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.742666    2564 manager.go:354] Recovery completed
Sep 10 23:01:34.748786 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.748689    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.750610 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.750593    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.750702 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.750620    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.750702 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.750631    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.751235 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751143    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 23:01:34.751235 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751155    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 23:01:34.751235 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751171    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:01:34.751235 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751226    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.751947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751929    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.752021 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751961    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.752021 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.751975    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.752332 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.752318    2564 policy_none.go:49] "None policy: Start"
Sep 10 23:01:34.752389 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.752338    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 23:01:34.752389 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.752351    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 23:01:34.752861 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.752790    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-219.ec2.internal.18d417f567e7eb37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-219.ec2.internal,UID:ip-10-0-129-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-219.ec2.internal,},FirstTimestamp:2026-09-10 23:01:34.750608183 +0000 UTC m=+0.382141104,LastTimestamp:2026-09-10 23:01:34.750608183 +0000 UTC m=+0.382141104,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-219.ec2.internal,}"
Sep 10 23:01:34.753535 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.753524    2564 policy_none.go:47] "Start"
Sep 10 23:01:34.777843 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.777830    2564 manager.go:337] "Starting Device Plugin manager"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.777860    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.777869    2564 server.go:87] "Starting device plugin registration server"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.778176    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.778207    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.778303    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.778378    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.778386    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.778929    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 23:01:34.788904 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.778975    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:34.844710 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.844680    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 23:01:34.845838 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.845821    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 23:01:34.845926 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.845875    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 23:01:34.845926 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.845902    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 23:01:34.846023 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.845975    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 23:01:34.848091 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.848073    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:34.878773 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.878757    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.881677 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.881661    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.881748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.881689    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.881748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.881700    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.881748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.881719    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-219.ec2.internal"
Sep 10 23:01:34.889924 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.889908    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-219.ec2.internal"
Sep 10 23:01:34.889977 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.889926    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-219.ec2.internal\": node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:34.904955 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.904899    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:34.946241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.946222    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"]
Sep 10 23:01:34.946314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.946277    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.947298 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.947282    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.947370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.947311    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.947370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.947322    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.950641 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.950628    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.950777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.950762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:34.950813 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.950792    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.951403 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.951389    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.951468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.951393    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.951468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.951439    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.951468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.951452    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.951555 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.951415    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.951555 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.951491    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.952544 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.952532    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:34.952596 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.952553    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:34.953162 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.953147    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:34.953268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.953174    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:34.953268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:34.953188    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:34.976798 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.976781    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-219.ec2.internal\" not found" node="ip-10-0-129-219.ec2.internal"
Sep 10 23:01:34.981115 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:34.981102    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-219.ec2.internal\" not found" node="ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.005479 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:35.005462    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:35.030635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.030617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fe73ced9a861e39eebd93ff0adf2c7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"7fe73ced9a861e39eebd93ff0adf2c7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.030693 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.030639    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fe73ced9a861e39eebd93ff0adf2c7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"7fe73ced9a861e39eebd93ff0adf2c7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.030693 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.030655    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/586716cd581743efb727a55187e31d8e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-219.ec2.internal\" (UID: \"586716cd581743efb727a55187e31d8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.105868 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:35.105849    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:35.131147 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.131131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fe73ced9a861e39eebd93ff0adf2c7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"7fe73ced9a861e39eebd93ff0adf2c7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.131225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.131163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fe73ced9a861e39eebd93ff0adf2c7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"7fe73ced9a861e39eebd93ff0adf2c7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.131225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.131181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/586716cd581743efb727a55187e31d8e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-219.ec2.internal\" (UID: \"586716cd581743efb727a55187e31d8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.131301 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.131134    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7fe73ced9a861e39eebd93ff0adf2c7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"7fe73ced9a861e39eebd93ff0adf2c7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.131301 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.131243    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fe73ced9a861e39eebd93ff0adf2c7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"7fe73ced9a861e39eebd93ff0adf2c7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.131301 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.131216    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/586716cd581743efb727a55187e31d8e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-219.ec2.internal\" (UID: \"586716cd581743efb727a55187e31d8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.206422 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:35.206366    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:35.279949 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.279922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.285277 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:35.285259    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:35.307252 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:35.307226    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:35.407789 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:35.407757    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:36.853745 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:36.853723    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:36.954300 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:36.954277    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:37.015746 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.015729    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 23:01:37.016095 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.015866    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:01:37.054971 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:37.054947    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:37.084443 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.084375    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 22:56:36 +0000 UTC" deadline="2028-06-08 01:53:19.408945492 +0000 UTC"
Sep 10 23:01:37.084443 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.084414    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15266h51m42.324535725s"
Sep 10 23:01:37.140792 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:37.140769    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe73ced9a861e39eebd93ff0adf2c7a.slice/crio-7074264eaa2be671645eca8c9cb8afb996b6a72908e569c0757dd08b831d0079 WatchSource:0}: Error finding container 7074264eaa2be671645eca8c9cb8afb996b6a72908e569c0757dd08b831d0079: Status 404 returned error can't find the container with id 7074264eaa2be671645eca8c9cb8afb996b6a72908e569c0757dd08b831d0079
Sep 10 23:01:37.141619 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:37.141599    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586716cd581743efb727a55187e31d8e.slice/crio-eff9bf7a95388027d58da05df8c7d577930f45b1df2b0d00432bd39fd2647ab2 WatchSource:0}: Error finding container eff9bf7a95388027d58da05df8c7d577930f45b1df2b0d00432bd39fd2647ab2: Status 404 returned error can't find the container with id eff9bf7a95388027d58da05df8c7d577930f45b1df2b0d00432bd39fd2647ab2
Sep 10 23:01:37.146568 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.146554    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:01:37.148738 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.148722    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:37.155461 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:37.155444    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Sep 10 23:01:37.194384 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.194332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal" event={"ID":"586716cd581743efb727a55187e31d8e","Type":"ContainerStarted","Data":"eff9bf7a95388027d58da05df8c7d577930f45b1df2b0d00432bd39fd2647ab2"}
Sep 10 23:01:37.195419 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.195400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" event={"ID":"7fe73ced9a861e39eebd93ff0adf2c7a","Type":"ContainerStarted","Data":"7074264eaa2be671645eca8c9cb8afb996b6a72908e569c0757dd08b831d0079"}
Sep 10 23:01:37.225814 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.225794    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:37.275532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.275514    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:37.285268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.285251    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:01:37.286999 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.286987    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Sep 10 23:01:37.296622 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.296606    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:01:37.589270 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:37.588924    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:38.055070 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.054997    2564 apiserver.go:52] "Watching apiserver"
Sep 10 23:01:38.061590 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.061548    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 23:01:38.062110 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.062070    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc","openshift-cluster-node-tuning-operator/tuned-v66qs","openshift-image-registry/node-ca-9k8bl","openshift-multus/multus-98f8k","openshift-multus/multus-additional-cni-plugins-xhqtb","openshift-network-diagnostics/network-check-target-gck4c","openshift-network-operator/iptables-alerter-v7whf","kube-system/global-pull-secret-syncer-psv9l","kube-system/konnectivity-agent-hwkpx","kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal","openshift-dns/node-resolver-grwkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal","openshift-multus/network-metrics-daemon-9vlkd","openshift-ovn-kubernetes/ovnkube-node-dq9mr"]
Sep 10 23:01:38.063915 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.063893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.065186 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.065164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.065280 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.065235    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:38.065950 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.065907    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78gpp\""
Sep 10 23:01:38.066177 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.066152    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.066395 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.066365    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 23:01:38.066395 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.066385    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.066525 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.066506    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.068642 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.068623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 23:01:38.069037 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.069017    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.069138 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.069040    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.069138 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.069083    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d9zbd\""
Sep 10 23:01:38.070013 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.069993    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.072214 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.071880    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.072214 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.072076    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 23:01:38.072548 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.072531    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 23:01:38.072628 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.072536    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.072628 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.072536    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k9sdg\""
Sep 10 23:01:38.074074 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.074052    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.075380 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.075362    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 23:01:38.075477 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.075449    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:38.075557 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.075530    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:38.076382 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.076364    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5ktjc\""
Sep 10 23:01:38.076571 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.076550    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 23:01:38.076667 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.076612    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 23:01:38.077257 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.077241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.078955 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.078922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.079658 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.079609    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v9pzx\""
Sep 10 23:01:38.079770 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.079755    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 23:01:38.080385 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.079960    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 23:01:38.080385 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.080170    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.080866 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.080847    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.081355 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.081336    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 23:01:38.081463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.081359    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.081463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.081443    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f4g4m\""
Sep 10 23:01:38.081830 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.081809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.082651 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.082635    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlw2p\""
Sep 10 23:01:38.082751 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.082674    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.082816 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.082773    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.083148 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.083132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.083795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.083779    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.083996 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.083962    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 23:01:38.085634 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.085173    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 23:01:38.085634 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.085258    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.085789 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.085651    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 23:01:38.085789 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.085663    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086011    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086069    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4ms9j\""
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086084    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5791960f-3106-459f-bfad-3ca157fbeec3-agent-certs\") pod \"konnectivity-agent-hwkpx\" (UID: \"5791960f-3106-459f-bfad-3ca157fbeec3\") " pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-cni-multus\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-conf-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086087    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-etc-kubernetes\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aacf4ed-66f3-49ab-a187-de54bc149343-host-slash\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086327    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5xg\" (UniqueName: \"kubernetes.io/projected/0aacf4ed-66f3-49ab-a187-de54bc149343-kube-api-access-lg5xg\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.086424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cnibin\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4g5xv\""
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-cni-bin\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-hostroot\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-multus-certs\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.086970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qt8\" (UniqueName: \"kubernetes.io/projected/6ae183f5-9da6-46d2-b227-78d5de9e33a5-kube-api-access-r2qt8\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-system-cni-dir\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-system-cni-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ac069be-2664-461a-8d5f-0cb231d3b4d9-host\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpcv\" (UniqueName: \"kubernetes.io/projected/347d9a42-6aa2-4615-b7fd-0ae132d0f512-kube-api-access-mvpcv\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5791960f-3106-459f-bfad-3ca157fbeec3-konnectivity-ca\") pod \"konnectivity-agent-hwkpx\" (UID: \"5791960f-3106-459f-bfad-3ca157fbeec3\") " pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-socket-dir-parent\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-kubelet\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-daemon-config\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.087354    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-cni-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-cnibin\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087466    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c049943-25e6-47aa-b615-1ac1c5707d84-cni-binary-copy\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.087490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-netns\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.088104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0aacf4ed-66f3-49ab-a187-de54bc149343-iptables-alerter-script\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.088104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ac069be-2664-461a-8d5f-0cb231d3b4d9-serviceca\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.088104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-k8s-cni-cncf-io\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.088104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slnd\" (UniqueName: \"kubernetes.io/projected/6c049943-25e6-47aa-b615-1ac1c5707d84-kube-api-access-5slnd\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.088104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.087645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tw25\" (UniqueName: \"kubernetes.io/projected/4ac069be-2664-461a-8d5f-0cb231d3b4d9-kube-api-access-5tw25\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.088654 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.088570    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:01:38.111526 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.111504    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2z2zx"
Sep 10 23:01:38.119376 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.119357    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2z2zx"
Sep 10 23:01:38.176224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.176199    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 23:01:38.188360 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-kubernetes\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.188479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-lib-modules\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.188479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4dm\" (UniqueName: \"kubernetes.io/projected/cb323308-f2ff-4cbc-a674-2eafb4880e45-kube-api-access-nm4dm\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.188479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-run-netns\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.188479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-node-log\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.188635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.188635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188517    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-env-overrides\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.188635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5xg\" (UniqueName: \"kubernetes.io/projected/0aacf4ed-66f3-49ab-a187-de54bc149343-kube-api-access-lg5xg\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.188635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.188635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysctl-d\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-tuned\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6nf\" (UniqueName: \"kubernetes.io/projected/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-kube-api-access-rf6nf\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-kubelet-config\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.188748    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cnibin\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-hostroot\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.188829    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:38.688805232 +0000 UTC m=+2.974826412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188828    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cnibin\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-hostroot\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-multus-certs\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-cni-netd\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.188896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-multus-certs\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-hosts-file\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghxqk\" (UniqueName: \"kubernetes.io/projected/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-kube-api-access-ghxqk\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.188984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-dbus\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qt8\" (UniqueName: \"kubernetes.io/projected/6ae183f5-9da6-46d2-b227-78d5de9e33a5-kube-api-access-r2qt8\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-system-cni-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189061    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysconfig\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ac069be-2664-461a-8d5f-0cb231d3b4d9-host\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189148    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-system-cni-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-daemon-config\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189202    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysctl-conf\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189208    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ac069be-2664-461a-8d5f-0cb231d3b4d9-host\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189229    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-socket-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-slash\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ac069be-2664-461a-8d5f-0cb231d3b4d9-serviceca\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.189523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-k8s-cni-cncf-io\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slnd\" (UniqueName: \"kubernetes.io/projected/6c049943-25e6-47aa-b615-1ac1c5707d84-kube-api-access-5slnd\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189434    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-k8s-cni-cncf-io\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-var-lib-kubelet\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-kubelet\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5791960f-3106-459f-bfad-3ca157fbeec3-agent-certs\") pod \"konnectivity-agent-hwkpx\" (UID: \"5791960f-3106-459f-bfad-3ca157fbeec3\") " pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-cni-multus\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-etc-kubernetes\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-modprobe-d\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ac069be-2664-461a-8d5f-0cb231d3b4d9-serviceca\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-daemon-config\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovn-node-metrics-cert\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aacf4ed-66f3-49ab-a187-de54bc149343-host-slash\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189809    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-etc-kubernetes\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.190507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-tmp\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-var-lib-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-cni-multus\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aacf4ed-66f3-49ab-a187-de54bc149343-host-slash\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-ovn\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-cni-bin\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j9k\" (UniqueName: \"kubernetes.io/projected/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-kube-api-access-k6j9k\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-cni-bin\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.189996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-sys-fs\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-systemd-units\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-cni-bin\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovnkube-config\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-system-cni-dir\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190060    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cni-binary-copy\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-host\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/347d9a42-6aa2-4615-b7fd-0ae132d0f512-system-cni-dir\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.191269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpcv\" (UniqueName: \"kubernetes.io/projected/347d9a42-6aa2-4615-b7fd-0ae132d0f512-kube-api-access-mvpcv\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5791960f-3106-459f-bfad-3ca157fbeec3-konnectivity-ca\") pod \"konnectivity-agent-hwkpx\" (UID: \"5791960f-3106-459f-bfad-3ca157fbeec3\") " pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-socket-dir-parent\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-kubelet\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190315    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-cni-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-cnibin\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-var-lib-kubelet\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c049943-25e6-47aa-b615-1ac1c5707d84-cni-binary-copy\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-socket-dir-parent\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-cni-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-cnibin\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-netns\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-device-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0aacf4ed-66f3-49ab-a187-de54bc149343-iptables-alerter-script\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.191899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190517    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-registration-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-host-run-netns\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190554    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-systemd\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovnkube-script-lib\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tw25\" (UniqueName: \"kubernetes.io/projected/4ac069be-2664-461a-8d5f-0cb231d3b4d9-kube-api-access-5tw25\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5791960f-3106-459f-bfad-3ca157fbeec3-konnectivity-ca\") pod \"konnectivity-agent-hwkpx\" (UID: \"5791960f-3106-459f-bfad-3ca157fbeec3\") " pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-systemd\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-run\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-sys\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-etc-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-log-socket\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-tmp-dir\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-conf-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c049943-25e6-47aa-b615-1ac1c5707d84-cni-binary-copy\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.190955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c049943-25e6-47aa-b615-1ac1c5707d84-multus-conf-dir\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.191032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0aacf4ed-66f3-49ab-a187-de54bc149343-iptables-alerter-script\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.192686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.191169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/347d9a42-6aa2-4615-b7fd-0ae132d0f512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.194290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.194269    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5791960f-3106-459f-bfad-3ca157fbeec3-agent-certs\") pod \"konnectivity-agent-hwkpx\" (UID: \"5791960f-3106-459f-bfad-3ca157fbeec3\") " pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.195230 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.195206    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:38.195230 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.195229    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:38.195378 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.195241    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:38.195378 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.195318    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:38.695301695 +0000 UTC m=+2.981322893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:38.197612 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.197590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5xg\" (UniqueName: \"kubernetes.io/projected/0aacf4ed-66f3-49ab-a187-de54bc149343-kube-api-access-lg5xg\") pod \"iptables-alerter-v7whf\" (UID: \"0aacf4ed-66f3-49ab-a187-de54bc149343\") " pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.198378 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.198359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qt8\" (UniqueName: \"kubernetes.io/projected/6ae183f5-9da6-46d2-b227-78d5de9e33a5-kube-api-access-r2qt8\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.198618 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.198599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slnd\" (UniqueName: \"kubernetes.io/projected/6c049943-25e6-47aa-b615-1ac1c5707d84-kube-api-access-5slnd\") pod \"multus-98f8k\" (UID: \"6c049943-25e6-47aa-b615-1ac1c5707d84\") " pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.198940 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.198920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpcv\" (UniqueName: \"kubernetes.io/projected/347d9a42-6aa2-4615-b7fd-0ae132d0f512-kube-api-access-mvpcv\") pod \"multus-additional-cni-plugins-xhqtb\" (UID: \"347d9a42-6aa2-4615-b7fd-0ae132d0f512\") " pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.199064 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.199046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tw25\" (UniqueName: \"kubernetes.io/projected/4ac069be-2664-461a-8d5f-0cb231d3b4d9-kube-api-access-5tw25\") pod \"node-ca-9k8bl\" (UID: \"4ac069be-2664-461a-8d5f-0cb231d3b4d9\") " pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.291649 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-tmp\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.291649 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-var-lib-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-ovn\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-cni-bin\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j9k\" (UniqueName: \"kubernetes.io/projected/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-kube-api-access-k6j9k\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-ovn\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-cni-bin\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291801    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-var-lib-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-sys-fs\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-systemd-units\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.291875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovnkube-config\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-sys-fs\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291910    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-systemd-units\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-host\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.291997    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-host\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-device-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-registration-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-device-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-systemd\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292076    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-registration-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovnkube-script-lib\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-systemd\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-run\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.292206 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-run-systemd\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-sys\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-etc-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-log-socket\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292221    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-systemd\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-tmp-dir\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-log-socket\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-sys\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-kubernetes\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-lib-modules\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4dm\" (UniqueName: \"kubernetes.io/projected/cb323308-f2ff-4cbc-a674-2eafb4880e45-kube-api-access-nm4dm\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-kubernetes\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-run\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-etc-openvswitch\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292344    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-run-netns\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-node-log\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-run-netns\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-node-log\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-env-overrides\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysctl-d\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-lib-modules\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-tuned\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6nf\" (UniqueName: \"kubernetes.io/projected/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-kube-api-access-rf6nf\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-tmp-dir\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-kubelet-config\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-kubelet-config\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-cni-netd\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysctl-d\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-cni-netd\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-hosts-file\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghxqk\" (UniqueName: \"kubernetes.io/projected/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-kube-api-access-ghxqk\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-dbus\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-hosts-file\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysconfig\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.293881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-dbus\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysconfig\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysctl-conf\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-socket-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.292985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-slash\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-var-lib-kubelet\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-kubelet\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293095    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-socket-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-modprobe-d\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovn-node-metrics-cert\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.293172    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293190    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovnkube-config\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.293233    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:01:38.793215492 +0000 UTC m=+3.079236692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293269    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-var-lib-kubelet\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.294775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovnkube-script-lib\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293397    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-kubelet\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293437    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb323308-f2ff-4cbc-a674-2eafb4880e45-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-sysctl-conf\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-modprobe-d\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-host-slash\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.293751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-env-overrides\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.294526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-tmp\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.294961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-etc-tuned\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.295598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.295386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-ovn-node-metrics-cert\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.301798 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.301776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j9k\" (UniqueName: \"kubernetes.io/projected/9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b-kube-api-access-k6j9k\") pod \"ovnkube-node-dq9mr\" (UID: \"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.301882 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.301846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4dm\" (UniqueName: \"kubernetes.io/projected/cb323308-f2ff-4cbc-a674-2eafb4880e45-kube-api-access-nm4dm\") pod \"aws-ebs-csi-driver-node-fkzkc\" (UID: \"cb323308-f2ff-4cbc-a674-2eafb4880e45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.302010 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.301992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6nf\" (UniqueName: \"kubernetes.io/projected/4eb6aed4-52f2-4bd2-997d-ba5b64a485ae-kube-api-access-rf6nf\") pod \"tuned-v66qs\" (UID: \"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae\") " pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.302010 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.302002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghxqk\" (UniqueName: \"kubernetes.io/projected/a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da-kube-api-access-ghxqk\") pod \"node-resolver-grwkz\" (UID: \"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da\") " pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.376047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.375984    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7whf"
Sep 10 23:01:38.383737 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.383714    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9k8bl"
Sep 10 23:01:38.393299 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.393276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-98f8k"
Sep 10 23:01:38.398967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.398941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xhqtb"
Sep 10 23:01:38.403663 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.403640    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:38.404727 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.404693    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:01:38.411305 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.411284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc"
Sep 10 23:01:38.418594 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.418578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v66qs"
Sep 10 23:01:38.426145 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.426127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:01:38.431736 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.431721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-grwkz"
Sep 10 23:01:38.695308 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.695230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:38.695460 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.695346    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:38.695460 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.695399    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:39.695384453 +0000 UTC m=+3.981405632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:38.795979 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.795947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:38.796132 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:38.796017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:38.796132 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.796105    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:38.796132 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.796114    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:38.796132 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.796130    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:38.796279 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.796141    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:38.796279 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.796174    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:01:39.796155568 +0000 UTC m=+4.082176762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:38.796279 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:38.796188    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:39.796182378 +0000 UTC m=+4.082203556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:39.112174 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.112028    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aacf4ed_66f3_49ab_a187_de54bc149343.slice/crio-04e9c92613d2bb77591c201421adfbcf09789ddb10cce0ffd5dd15525b80b459 WatchSource:0}: Error finding container 04e9c92613d2bb77591c201421adfbcf09789ddb10cce0ffd5dd15525b80b459: Status 404 returned error can't find the container with id 04e9c92613d2bb77591c201421adfbcf09789ddb10cce0ffd5dd15525b80b459
Sep 10 23:01:39.113779 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.113667    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5791960f_3106_459f_bfad_3ca157fbeec3.slice/crio-3a057599767f3ec655aff109b93088865ac602d6eb5aaef5f8fc81038d0bd8fd WatchSource:0}: Error finding container 3a057599767f3ec655aff109b93088865ac602d6eb5aaef5f8fc81038d0bd8fd: Status 404 returned error can't find the container with id 3a057599767f3ec655aff109b93088865ac602d6eb5aaef5f8fc81038d0bd8fd
Sep 10 23:01:39.116813 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.116785    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347d9a42_6aa2_4615_b7fd_0ae132d0f512.slice/crio-c025b47ab3afcdd803f75a9c46fe0ff861b1b3f9a462947eb3975191bdbe2dce WatchSource:0}: Error finding container c025b47ab3afcdd803f75a9c46fe0ff861b1b3f9a462947eb3975191bdbe2dce: Status 404 returned error can't find the container with id c025b47ab3afcdd803f75a9c46fe0ff861b1b3f9a462947eb3975191bdbe2dce
Sep 10 23:01:39.117762 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.117696    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac069be_2664_461a_8d5f_0cb231d3b4d9.slice/crio-7554f7730e3f07a7b97cfb33bc91b05b9222e67482cf8c4ff46a7a26acce681a WatchSource:0}: Error finding container 7554f7730e3f07a7b97cfb33bc91b05b9222e67482cf8c4ff46a7a26acce681a: Status 404 returned error can't find the container with id 7554f7730e3f07a7b97cfb33bc91b05b9222e67482cf8c4ff46a7a26acce681a
Sep 10 23:01:39.118809 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.118786    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb323308_f2ff_4cbc_a674_2eafb4880e45.slice/crio-4a29b3911fff2ee927252a352568502280f70dd5dace74261b3b682d182eba72 WatchSource:0}: Error finding container 4a29b3911fff2ee927252a352568502280f70dd5dace74261b3b682d182eba72: Status 404 returned error can't find the container with id 4a29b3911fff2ee927252a352568502280f70dd5dace74261b3b682d182eba72
Sep 10 23:01:39.119506 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.119483    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c049943_25e6_47aa_b615_1ac1c5707d84.slice/crio-45be5a4ac29c05d7c0c15662e7ca08ec8cc972914f05634af8e3b17bf0768085 WatchSource:0}: Error finding container 45be5a4ac29c05d7c0c15662e7ca08ec8cc972914f05634af8e3b17bf0768085: Status 404 returned error can't find the container with id 45be5a4ac29c05d7c0c15662e7ca08ec8cc972914f05634af8e3b17bf0768085
Sep 10 23:01:39.120227 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.120120    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf73bfe_33fb_41ca_81c9_a7a1e58bc56b.slice/crio-5cc5e21599037994ce84f0020d51cd979caa0ae82dfd30d2d9c715200eab7390 WatchSource:0}: Error finding container 5cc5e21599037994ce84f0020d51cd979caa0ae82dfd30d2d9c715200eab7390: Status 404 returned error can't find the container with id 5cc5e21599037994ce84f0020d51cd979caa0ae82dfd30d2d9c715200eab7390
Sep 10 23:01:39.121745 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.121722    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb6aed4_52f2_4bd2_997d_ba5b64a485ae.slice/crio-043396938c06532fc578f595451c30b37bed2b9b72ae31ec75a0e2f119e3172f WatchSource:0}: Error finding container 043396938c06532fc578f595451c30b37bed2b9b72ae31ec75a0e2f119e3172f: Status 404 returned error can't find the container with id 043396938c06532fc578f595451c30b37bed2b9b72ae31ec75a0e2f119e3172f
Sep 10 23:01:39.122742 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.122495    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:56:38 +0000 UTC" deadline="2028-02-09 02:42:16.498306569 +0000 UTC"
Sep 10 23:01:39.122742 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.122519    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12387h40m37.375791445s"
Sep 10 23:01:39.123140 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:01:39.123073    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17e0c8e_f203_4b9f_a43d_8c5e2f8f64da.slice/crio-7cd4e35bf359e6e4531f3f3d1f4b7bb43ab091590acf58ab7d1c3be11db8aad5 WatchSource:0}: Error finding container 7cd4e35bf359e6e4531f3f3d1f4b7bb43ab091590acf58ab7d1c3be11db8aad5: Status 404 returned error can't find the container with id 7cd4e35bf359e6e4531f3f3d1f4b7bb43ab091590acf58ab7d1c3be11db8aad5
Sep 10 23:01:39.199871 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.199836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v66qs" event={"ID":"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae","Type":"ContainerStarted","Data":"043396938c06532fc578f595451c30b37bed2b9b72ae31ec75a0e2f119e3172f"}
Sep 10 23:01:39.201598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.201417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"5cc5e21599037994ce84f0020d51cd979caa0ae82dfd30d2d9c715200eab7390"}
Sep 10 23:01:39.205511 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.205485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc" event={"ID":"cb323308-f2ff-4cbc-a674-2eafb4880e45","Type":"ContainerStarted","Data":"4a29b3911fff2ee927252a352568502280f70dd5dace74261b3b682d182eba72"}
Sep 10 23:01:39.206667 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.206641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grwkz" event={"ID":"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da","Type":"ContainerStarted","Data":"7cd4e35bf359e6e4531f3f3d1f4b7bb43ab091590acf58ab7d1c3be11db8aad5"}
Sep 10 23:01:39.207725 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.207693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-98f8k" event={"ID":"6c049943-25e6-47aa-b615-1ac1c5707d84","Type":"ContainerStarted","Data":"45be5a4ac29c05d7c0c15662e7ca08ec8cc972914f05634af8e3b17bf0768085"}
Sep 10 23:01:39.208918 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.208897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9k8bl" event={"ID":"4ac069be-2664-461a-8d5f-0cb231d3b4d9","Type":"ContainerStarted","Data":"7554f7730e3f07a7b97cfb33bc91b05b9222e67482cf8c4ff46a7a26acce681a"}
Sep 10 23:01:39.210093 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.210070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerStarted","Data":"c025b47ab3afcdd803f75a9c46fe0ff861b1b3f9a462947eb3975191bdbe2dce"}
Sep 10 23:01:39.211245 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.211222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hwkpx" event={"ID":"5791960f-3106-459f-bfad-3ca157fbeec3","Type":"ContainerStarted","Data":"3a057599767f3ec655aff109b93088865ac602d6eb5aaef5f8fc81038d0bd8fd"}
Sep 10 23:01:39.212767 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.212746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7whf" event={"ID":"0aacf4ed-66f3-49ab-a187-de54bc149343","Type":"ContainerStarted","Data":"04e9c92613d2bb77591c201421adfbcf09789ddb10cce0ffd5dd15525b80b459"}
Sep 10 23:01:39.702074 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.701961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:39.702227 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.702100    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:39.702227 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.702154    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:41.702136862 +0000 UTC m=+5.988158058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:39.802488 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.802332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:39.802488 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:39.802394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:39.802488 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.802472    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:39.802784 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.802528    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:01:41.802511743 +0000 UTC m=+6.088532937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:39.802784 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.802567    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:39.802784 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.802595    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:39.802784 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.802608    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:39.802784 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:39.802655    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:41.802639987 +0000 UTC m=+6.088661179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:40.122795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.122672    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:56:38 +0000 UTC" deadline="2028-02-08 06:28:49.648517428 +0000 UTC"
Sep 10 23:01:40.122795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.122748    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12367h27m9.525774088s"
Sep 10 23:01:40.194772 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.194504    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:40.194772 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:40.194637    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:40.194772 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.194735    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:40.195093 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:40.194811    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:40.195093 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.194870    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:40.195093 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:40.194942    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:40.241837 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.241666    2564 generic.go:356] "Generic (PLEG): container finished" podID="7fe73ced9a861e39eebd93ff0adf2c7a" containerID="c6fc4d19285255ab8cade41849669b14c2c2360beb4129787df048503d69413c" exitCode=0
Sep 10 23:01:40.241837 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.241781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" event={"ID":"7fe73ced9a861e39eebd93ff0adf2c7a","Type":"ContainerDied","Data":"c6fc4d19285255ab8cade41849669b14c2c2360beb4129787df048503d69413c"}
Sep 10 23:01:40.251994 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:40.251570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal" event={"ID":"586716cd581743efb727a55187e31d8e","Type":"ContainerStarted","Data":"7ad58e4ba2f285b60218e8e82ff262ec5a25de6e08f8aaab80ef1e7fac9195d4"}
Sep 10 23:01:41.271676 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:41.271482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" event={"ID":"7fe73ced9a861e39eebd93ff0adf2c7a","Type":"ContainerStarted","Data":"da321a29d155099105ce8eb024f56061d7aac10e36cd3bf14c1366765ad30d21"}
Sep 10 23:01:41.288892 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:41.288798    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal" podStartSLOduration=4.288778378 podStartE2EDuration="4.288778378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:40.276221074 +0000 UTC m=+4.562242276" watchObservedRunningTime="2026-09-10 23:01:41.288778378 +0000 UTC m=+5.574799581"
Sep 10 23:01:41.731526 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:41.730931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:41.731526 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.731115    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:41.731526 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.731177    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:45.731159706 +0000 UTC m=+10.017180908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:41.831537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:41.831596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.831751    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.831770    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.831783    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.831842    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:45.831821714 +0000 UTC m=+10.117842915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.832228    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:41.832312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:41.832279    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:01:45.832263537 +0000 UTC m=+10.118284738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:42.192413 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:42.192339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:42.192570 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:42.192522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:42.192570 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:42.192527    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:42.192676 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:42.192658    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:42.192783 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:42.192766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:42.192885 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:42.192848    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:44.193046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:44.192245    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:44.193046 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:44.192378    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:44.193046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:44.192761    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:44.193046 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:44.192856    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:44.193046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:44.192932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:44.193046 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:44.193019    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:45.765455 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:45.765425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:45.765965 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.765561    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:45.765965 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.765623    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:53.76560602 +0000 UTC m=+18.051627198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:45.866022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:45.866083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.866231    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.866254    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.866267    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.866322    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:01:53.866303271 +0000 UTC m=+18.152324453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.866547    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:45.866673 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:45.866606    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:01:53.866590366 +0000 UTC m=+18.152611558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:46.193219 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:46.192929    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:46.193219 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:46.193078    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:46.193219 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:46.193095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:46.193219 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:46.193088    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:46.193219 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:46.193177    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:46.193565 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:46.193257    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:48.191930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:48.191893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:48.191930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:48.191929    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:48.192443 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:48.191898    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:48.192443 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:48.192047    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:48.192443 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:48.192170    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:48.192443 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:48.192259    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:50.192633 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:50.192590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:50.193109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:50.192596    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:50.193109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:50.192730    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:50.193109 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:50.192724    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:50.193109 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:50.192823    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:50.193109 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:50.192926    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:52.192298 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:52.192100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:52.192765 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:52.192100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:52.192765 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:52.192383    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:52.192765 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:52.192129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:52.192765 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:52.192454    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:52.192765 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:52.192570    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:53.829043 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:53.828934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:53.829463 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.829118    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:53.829463 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.829183    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.829167625 +0000 UTC m=+34.115188827 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:53.929672 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:53.929640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:53.929858 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:53.929682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:53.929858 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.929817    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:53.929858 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.929832    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:53.929858 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.929849    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:53.929858 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.929861    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:53.930048 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.929891    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.92986237 +0000 UTC m=+34.215883564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:53.930048 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:53.929912    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.929900292 +0000 UTC m=+34.215921484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:54.192639 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:54.192570    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:54.192639 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:54.192603    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:54.192873 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:54.192573    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:54.192873 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:54.192730    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:54.192978 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:54.192895    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:54.193032 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:54.192999    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:56.192720 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:56.192675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:56.193148 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:56.192809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:56.193148 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:56.192826    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:56.193148 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:56.192909    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:56.193148 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:56.192974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:56.193148 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:56.193064    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:57.307297 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.306608    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-grwkz" event={"ID":"a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da","Type":"ContainerStarted","Data":"d79970a057eac7cfa9dd48de0acee93cdd723b3da4a9d520a35d436255aa32a5"}
Sep 10 23:01:57.309095 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.309030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9k8bl" event={"ID":"4ac069be-2664-461a-8d5f-0cb231d3b4d9","Type":"ContainerStarted","Data":"c05f7d25a36b9e4f72e2fa243918aa6f00586fe125d53f09bfb5fbcb308556b5"}
Sep 10 23:01:57.311414 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.311363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerStarted","Data":"b39e7d07e1a5ea2b1f62b825ef46f79a30912f3339ecfaf73f923637f3beaf39"}
Sep 10 23:01:57.313907 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.313825    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hwkpx" event={"ID":"5791960f-3106-459f-bfad-3ca157fbeec3","Type":"ContainerStarted","Data":"8abed341e61decd53ca24e2bb2e93c0269936e645607bdcd1245186770cfbfc3"}
Sep 10 23:01:57.317449 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.317400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v66qs" event={"ID":"4eb6aed4-52f2-4bd2-997d-ba5b64a485ae","Type":"ContainerStarted","Data":"8a5e487c7c68b7b5fe1c26da683a2ca4fc0e4c92ad0362485407bf2571c6ad6d"}
Sep 10 23:01:57.321548 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.321456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"1343e8dbd59390667f8020cdf06b63d1c383cb9a88f9eb5f5c096b238fdc0d21"}
Sep 10 23:01:57.322697 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.322350    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" podStartSLOduration=20.322334519 podStartE2EDuration="20.322334519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:41.289877721 +0000 UTC m=+5.575898924" watchObservedRunningTime="2026-09-10 23:01:57.322334519 +0000 UTC m=+21.608355721"
Sep 10 23:01:57.323984 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.323930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc" event={"ID":"cb323308-f2ff-4cbc-a674-2eafb4880e45","Type":"ContainerStarted","Data":"08c68e728c4432d43c3415903dab3ac68d98e1ef9d8cf1c6af8de837c7a670ad"}
Sep 10 23:01:57.337266 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.337223    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9k8bl" podStartSLOduration=3.453115971 podStartE2EDuration="21.337210075s" totalImagesPullingTime="17.884094104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:57.337051373 +0000 UTC m=+21.623072597" watchObservedRunningTime="2026-09-10 23:01:57.337210075 +0000 UTC m=+21.623231275"
Sep 10 23:01:57.337565 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.337539    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-grwkz" podStartSLOduration=3.44134638 podStartE2EDuration="21.337534108s" totalImagesPullingTime="17.896187728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:57.322520537 +0000 UTC m=+21.608541740" watchObservedRunningTime="2026-09-10 23:01:57.337534108 +0000 UTC m=+21.623555309"
Sep 10 23:01:57.389873 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.389825    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hwkpx" podStartSLOduration=7.0617456690000004 podStartE2EDuration="21.38980962s" totalImagesPullingTime="14.328063951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:57.369351482 +0000 UTC m=+21.655372684" watchObservedRunningTime="2026-09-10 23:01:57.38980962 +0000 UTC m=+21.675830822"
Sep 10 23:01:57.390225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.390197    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v66qs" podStartSLOduration=3.488217791 podStartE2EDuration="21.390187359s" totalImagesPullingTime="17.901969568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:57.389638041 +0000 UTC m=+21.675659252" watchObservedRunningTime="2026-09-10 23:01:57.390187359 +0000 UTC m=+21.676208560"
Sep 10 23:01:57.448623 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.448590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:57.532607 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:57.532511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.048179 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.048093    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.148840 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.148778    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.191836 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.191807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:01:58.192037 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:58.191953    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:01:58.192141 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.192061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:01:58.192209 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:58.192184    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:01:58.192262 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.192251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:01:58.192366 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:01:58.192348    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:01:58.326742 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.326693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7whf" event={"ID":"0aacf4ed-66f3-49ab-a187-de54bc149343","Type":"ContainerStarted","Data":"e9f2c504a1a77fd991e9d76632036e0043e3811ad9cdc99aca3c94d59ca8be1c"}
Sep 10 23:01:58.329747 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.329723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"c5c0289ddc53ee5d1bb34293e2c1f23c4fa82fc0546518fd068934e6c9dc3871"}
Sep 10 23:01:58.329856 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.329787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"773753a873236ce01f9839d5e197ca4763b57abffd6a2eb37830c5a96d17c042"}
Sep 10 23:01:58.329856 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.329848    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"cd59ca2b9b383f6dacc693239ef697b0f1f943e4847a8162b41116288bb70581"}
Sep 10 23:01:58.329961 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.329889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"c87bd383160b2d2f39d4f74e38c4d1aebc59d15fa776e527fb507ef9b4bfafa6"}
Sep 10 23:01:58.329961 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.329919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"f7aa991a8f4fa47e66deafef0b03766a4683ec6e3d4abcabc10ad73f6d8d2eb9"}
Sep 10 23:01:58.331066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.331046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-98f8k" event={"ID":"6c049943-25e6-47aa-b615-1ac1c5707d84","Type":"ContainerStarted","Data":"00f3332742f6f926be35df57866bce23d3a92511ae964e0003253fb71af6d8a4"}
Sep 10 23:01:58.332386 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.332362    2564 generic.go:356] "Generic (PLEG): container finished" podID="347d9a42-6aa2-4615-b7fd-0ae132d0f512" containerID="b39e7d07e1a5ea2b1f62b825ef46f79a30912f3339ecfaf73f923637f3beaf39" exitCode=0
Sep 10 23:01:58.332491 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.332456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerDied","Data":"b39e7d07e1a5ea2b1f62b825ef46f79a30912f3339ecfaf73f923637f3beaf39"}
Sep 10 23:01:58.341199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.341164    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v7whf" podStartSLOduration=4.451790829 podStartE2EDuration="22.341153851s" totalImagesPullingTime="17.889363022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:58.340967069 +0000 UTC m=+22.626988270" watchObservedRunningTime="2026-09-10 23:01:58.341153851 +0000 UTC m=+22.627175049"
Sep 10 23:01:58.345360 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.345342    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.356469 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.356435    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-98f8k" podStartSLOduration=4.223479181 podStartE2EDuration="22.356423532s" totalImagesPullingTime="18.132944351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:58.356216853 +0000 UTC m=+22.642238054" watchObservedRunningTime="2026-09-10 23:01:58.356423532 +0000 UTC m=+22.642444732"
Sep 10 23:01:58.403586 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.403563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.480615 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.480590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.582587 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.582563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:58.639733 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.639689    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 23:01:58.682613 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:58.682590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:59.133938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.133799    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:01:58.639720849Z","UUID":"79dcd080-17e4-4586-9ce9-0d92435bb7b7","Handler":null,"Name":"","Endpoint":""}
Sep 10 23:01:59.135971 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.135933    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 23:01:59.136215 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.136178    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 23:01:59.223097 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.223065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:59.324417 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.324384    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:59.337692 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.337661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc" event={"ID":"cb323308-f2ff-4cbc-a674-2eafb4880e45","Type":"ContainerStarted","Data":"e7341b9b8122fe8adb8b8f290fe6ba625330afced4cd4a4d95e695c747530fa0"}
Sep 10 23:01:59.641386 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.641173    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:59.741972 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:01:59.741938    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:00.192582 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:00.192547    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:00.192582 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:00.192579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:00.192776 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:00.192679    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:00.192776 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:00.192697    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:02:00.192885 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:00.192829    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:02:00.192939 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:00.192912    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:02:00.343735 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:00.343646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"03a9983e4eefb30d6e3dbd5521e8ecbf3143db55293f00ff45072664d7e5ed60"}
Sep 10 23:02:00.346053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:00.346023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc" event={"ID":"cb323308-f2ff-4cbc-a674-2eafb4880e45","Type":"ContainerStarted","Data":"64cbec88f42a998570b9ac29f11d89a4de6c2a0546978fdf109ae8c7e45fa25b"}
Sep 10 23:02:00.365589 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:00.365546    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkzkc" podStartSLOduration=3.796689792 podStartE2EDuration="24.365533268s" totalImagesPullingTime="20.568843476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:00.364983224 +0000 UTC m=+24.651004415" watchObservedRunningTime="2026-09-10 23:02:00.365533268 +0000 UTC m=+24.651554516"
Sep 10 23:02:01.027282 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:01.027228    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:01.126230 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:01.126193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:01.568381 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:01.568356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:02:01.569062 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:01.569040    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:02:02.192756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:02.192668    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:02.192756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:02.192667    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:02.192756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:02.192726    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:02.193016 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:02.192827    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:02:02.193155 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:02.193120    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:02:02.193239 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:02.193197    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:02:02.350714 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:02.350675    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:02:02.351247 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:02.351219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hwkpx"
Sep 10 23:02:02.418720 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:02.418686    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:03.357938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:03.357250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" event={"ID":"9bf73bfe-33fb-41ca-81c9-a7a1e58bc56b","Type":"ContainerStarted","Data":"078b9b6e0f0efbb724925d5e017ca316e9ac14227e29b1f484b2e88687ed9cae"}
Sep 10 23:02:03.357938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:03.357651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:02:03.357938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:03.357766    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:02:03.376892 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:03.376866    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:02:03.377104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:03.377086    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:02:03.393521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:03.390309    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr" podStartSLOduration=9.453427653 podStartE2EDuration="27.390292914s" totalImagesPullingTime="17.936865261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:03.39023141 +0000 UTC m=+27.676252604" watchObservedRunningTime="2026-09-10 23:02:03.390292914 +0000 UTC m=+27.676314107"
Sep 10 23:02:04.192463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.192429    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:04.192463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.192454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:04.192728 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:04.192531    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:02:04.192728 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.192622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:04.192728 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:04.192619    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:02:04.192728 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:04.192686    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:02:04.363607 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.363568    2564 generic.go:356] "Generic (PLEG): container finished" podID="347d9a42-6aa2-4615-b7fd-0ae132d0f512" containerID="809e3e9fcf1c4173a85666cbb8a10801300f40364c380b1b1538ff1aeff78312" exitCode=0
Sep 10 23:02:04.364177 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.363807    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 23:02:04.365344 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.365310    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerDied","Data":"809e3e9fcf1c4173a85666cbb8a10801300f40364c380b1b1538ff1aeff78312"}
Sep 10 23:02:04.959261 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.959233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psv9l"]
Sep 10 23:02:04.959413 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.959344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:04.959479 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:04.959448    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:02:04.961913 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.961849    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gck4c"]
Sep 10 23:02:04.962028 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.961985    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:04.962140 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:04.962105    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:02:04.962565 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.962509    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vlkd"]
Sep 10 23:02:04.962659 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:04.962594    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:04.962739 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:04.962682    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:02:05.367576 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:05.367502    2564 generic.go:356] "Generic (PLEG): container finished" podID="347d9a42-6aa2-4615-b7fd-0ae132d0f512" containerID="4d3a549d3322dc9603d6bed62edf9ad485eaa936e3544b74f559e2e3c5dae4ee" exitCode=0
Sep 10 23:02:05.368298 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:05.367584    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerDied","Data":"4d3a549d3322dc9603d6bed62edf9ad485eaa936e3544b74f559e2e3c5dae4ee"}
Sep 10 23:02:05.368298 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:05.367673    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 23:02:06.193329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:06.193093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:06.193455 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:06.193422    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:02:06.372776 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:06.372676    2564 generic.go:356] "Generic (PLEG): container finished" podID="347d9a42-6aa2-4615-b7fd-0ae132d0f512" containerID="55ea25ffe63cfe1acdfa593ad3a8327a8d121832ff3bc20a8cbbcdd63bb610b7" exitCode=0
Sep 10 23:02:06.372776 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:06.372737    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerDied","Data":"55ea25ffe63cfe1acdfa593ad3a8327a8d121832ff3bc20a8cbbcdd63bb610b7"}
Sep 10 23:02:07.192642 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:07.192596    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:07.192810 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:07.192700    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:02:07.192810 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:07.192763    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:07.192926 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:07.192874    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:02:07.274954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:07.274928    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:02:08.191956 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:08.191923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:08.192344 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:08.192043    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gck4c" podUID="640ae5d2-990e-45ac-b74b-869f5f837a2c"
Sep 10 23:02:09.192323 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.192294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:09.192869 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.192308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:09.192869 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.192421    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vlkd" podUID="6ae183f5-9da6-46d2-b227-78d5de9e33a5"
Sep 10 23:02:09.192869 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.192466    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psv9l" podUID="9b53e2c5-97cd-4de1-a418-40e5fe116a6f"
Sep 10 23:02:09.853175 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.853140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:09.853334 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.853282    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:02:09.853397 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.853350    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs podName:6ae183f5-9da6-46d2-b227-78d5de9e33a5 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:41.853329626 +0000 UTC m=+66.139350810 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs") pod "network-metrics-daemon-9vlkd" (UID: "6ae183f5-9da6-46d2-b227-78d5de9e33a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:02:09.953486 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.953458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:09.953616 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.953544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:09.953683 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.953642    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:02:09.953683 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.953656    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:02:09.953683 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.953662    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:02:09.953683 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.953676    2564 projected.go:196] Error preparing data for projected volume kube-api-access-fbhgr for pod openshift-network-diagnostics/network-check-target-gck4c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:02:09.953894 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.953752    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret podName:9b53e2c5-97cd-4de1-a418-40e5fe116a6f nodeName:}" failed. No retries permitted until 2026-09-10 23:02:41.953730201 +0000 UTC m=+66.239751398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret") pod "global-pull-secret-syncer-psv9l" (UID: "9b53e2c5-97cd-4de1-a418-40e5fe116a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:02:09.953894 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:09.953773    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr podName:640ae5d2-990e-45ac-b74b-869f5f837a2c nodeName:}" failed. No retries permitted until 2026-09-10 23:02:41.953763677 +0000 UTC m=+66.239784870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbhgr" (UniqueName: "kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr") pod "network-check-target-gck4c" (UID: "640ae5d2-990e-45ac-b74b-869f5f837a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:02:09.977363 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.977344    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeReady"
Sep 10 23:02:09.977490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:09.977477    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 23:02:10.020375 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.020345    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-srh7k"]
Sep 10 23:02:10.062173 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.062149    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6685x"]
Sep 10 23:02:10.062329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.062312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.064591 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.064562    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 23:02:10.064733 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.064654    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 23:02:10.064733 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.064682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tqzlq\""
Sep 10 23:02:10.064852 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.064807    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 23:02:10.082114 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.082086    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srh7k"]
Sep 10 23:02:10.082216 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.082123    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6685x"]
Sep 10 23:02:10.082216 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.082156    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8szzn"]
Sep 10 23:02:10.082216 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.082194    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.084624 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.084606    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 23:02:10.084903 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.084867    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 23:02:10.084903 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.084867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 23:02:10.084903 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.084890    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w4w8p\""
Sep 10 23:02:10.085108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.084914    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 23:02:10.104658 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.104640    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8szzn"]
Sep 10 23:02:10.104762 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.104745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.106987 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.106970    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 23:02:10.107370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.107189    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 23:02:10.107370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.107254    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-497ss\""
Sep 10 23:02:10.191956 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.191895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:10.196178 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.196154    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:02:10.196493 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.196357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcwbm\""
Sep 10 23:02:10.196550 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.196500    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:02:10.255775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.255750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6lx\" (UniqueName: \"kubernetes.io/projected/512332f9-1be2-462f-87db-a27b8c499f33-kube-api-access-4c6lx\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.255955 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.255797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.255955 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.255825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xh2j\" (UniqueName: \"kubernetes.io/projected/b9bed0bc-550e-4da3-9a09-223ff72d236e-kube-api-access-6xh2j\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.255955 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.255861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/512332f9-1be2-462f-87db-a27b8c499f33-data-volume\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.255955 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.255881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.256127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.255963    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bed0bc-550e-4da3-9a09-223ff72d236e-config-volume\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.256127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.256024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/512332f9-1be2-462f-87db-a27b8c499f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.256127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.256047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e99ce852-5695-4da2-a352-5d714874db49-cert\") pod \"ingress-canary-srh7k\" (UID: \"e99ce852-5695-4da2-a352-5d714874db49\") " pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.256127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.256079    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/512332f9-1be2-462f-87db-a27b8c499f33-crio-socket\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.256301 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.256156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bed0bc-550e-4da3-9a09-223ff72d236e-tmp-dir\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.256301 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.256177    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8f5\" (UniqueName: \"kubernetes.io/projected/e99ce852-5695-4da2-a352-5d714874db49-kube-api-access-gt8f5\") pod \"ingress-canary-srh7k\" (UID: \"e99ce852-5695-4da2-a352-5d714874db49\") " pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.356477 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bed0bc-550e-4da3-9a09-223ff72d236e-tmp-dir\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.356619 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8f5\" (UniqueName: \"kubernetes.io/projected/e99ce852-5695-4da2-a352-5d714874db49-kube-api-access-gt8f5\") pod \"ingress-canary-srh7k\" (UID: \"e99ce852-5695-4da2-a352-5d714874db49\") " pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.356619 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6lx\" (UniqueName: \"kubernetes.io/projected/512332f9-1be2-462f-87db-a27b8c499f33-kube-api-access-4c6lx\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.356619 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.356815 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xh2j\" (UniqueName: \"kubernetes.io/projected/b9bed0bc-550e-4da3-9a09-223ff72d236e-kube-api-access-6xh2j\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.356815 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.356640    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:10.356815 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/512332f9-1be2-462f-87db-a27b8c499f33-data-volume\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.356815 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.356720    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls podName:512332f9-1be2-462f-87db-a27b8c499f33 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:10.856681955 +0000 UTC m=+35.142703153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6685x" (UID: "512332f9-1be2-462f-87db-a27b8c499f33") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:10.356815 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.356815 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bed0bc-550e-4da3-9a09-223ff72d236e-config-volume\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/512332f9-1be2-462f-87db-a27b8c499f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9bed0bc-550e-4da3-9a09-223ff72d236e-tmp-dir\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.356964    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e99ce852-5695-4da2-a352-5d714874db49-cert\") pod \"ingress-canary-srh7k\" (UID: \"e99ce852-5695-4da2-a352-5d714874db49\") " pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.356964    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/512332f9-1be2-462f-87db-a27b8c499f33-data-volume\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.357037    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls podName:b9bed0bc-550e-4da3-9a09-223ff72d236e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:10.857011577 +0000 UTC m=+35.143032761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls") pod "dns-default-8szzn" (UID: "b9bed0bc-550e-4da3-9a09-223ff72d236e") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:10.357109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.357081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/512332f9-1be2-462f-87db-a27b8c499f33-crio-socket\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.357439 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.357311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/512332f9-1be2-462f-87db-a27b8c499f33-crio-socket\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.357439 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.357354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/512332f9-1be2-462f-87db-a27b8c499f33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.357550 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.357508    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bed0bc-550e-4da3-9a09-223ff72d236e-config-volume\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.361565 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.361541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e99ce852-5695-4da2-a352-5d714874db49-cert\") pod \"ingress-canary-srh7k\" (UID: \"e99ce852-5695-4da2-a352-5d714874db49\") " pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.368185 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.368156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6lx\" (UniqueName: \"kubernetes.io/projected/512332f9-1be2-462f-87db-a27b8c499f33-kube-api-access-4c6lx\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.368549 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.368524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8f5\" (UniqueName: \"kubernetes.io/projected/e99ce852-5695-4da2-a352-5d714874db49-kube-api-access-gt8f5\") pod \"ingress-canary-srh7k\" (UID: \"e99ce852-5695-4da2-a352-5d714874db49\") " pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.368549 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.368546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xh2j\" (UniqueName: \"kubernetes.io/projected/b9bed0bc-550e-4da3-9a09-223ff72d236e-kube-api-access-6xh2j\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.372403 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.372373    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srh7k"
Sep 10 23:02:10.860985 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.860943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:10.860985 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.860997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:10.861225 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.861121    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:10.861225 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.861140    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:10.861225 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.861188    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls podName:512332f9-1be2-462f-87db-a27b8c499f33 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:11.861170484 +0000 UTC m=+36.147191664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6685x" (UID: "512332f9-1be2-462f-87db-a27b8c499f33") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:10.861225 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:10.861203    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls podName:b9bed0bc-550e-4da3-9a09-223ff72d236e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:11.861196667 +0000 UTC m=+36.147217845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls") pod "dns-default-8szzn" (UID: "b9bed0bc-550e-4da3-9a09-223ff72d236e") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:10.864274 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:10.864250    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grwkz_a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da/dns-node-resolver/0.log"
Sep 10 23:02:11.192176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.192038    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:11.192294 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.192079    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:11.194727 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.194693    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:02:11.195465 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.195446    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-glltp\""
Sep 10 23:02:11.195574 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.195474    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:02:11.264211 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.264194    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9k8bl_4ac069be-2664-461a-8d5f-0cb231d3b4d9/node-ca/0.log"
Sep 10 23:02:11.387793 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.387772    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srh7k"]
Sep 10 23:02:11.391297 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:11.391277    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99ce852_5695_4da2_a352_5d714874db49.slice/crio-2a29b22495f596ed83e0c143f4bf993bfa92f5b2d5da0d31df81e1c2f462fe7a WatchSource:0}: Error finding container 2a29b22495f596ed83e0c143f4bf993bfa92f5b2d5da0d31df81e1c2f462fe7a: Status 404 returned error can't find the container with id 2a29b22495f596ed83e0c143f4bf993bfa92f5b2d5da0d31df81e1c2f462fe7a
Sep 10 23:02:11.867286 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.867222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:11.867286 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:11.867260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:11.867440 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:11.867348    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:11.867440 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:11.867356    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:11.867440 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:11.867397    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls podName:b9bed0bc-550e-4da3-9a09-223ff72d236e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:13.867384647 +0000 UTC m=+38.153405825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls") pod "dns-default-8szzn" (UID: "b9bed0bc-550e-4da3-9a09-223ff72d236e") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:11.867440 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:11.867410    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls podName:512332f9-1be2-462f-87db-a27b8c499f33 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:13.867404196 +0000 UTC m=+38.153425374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6685x" (UID: "512332f9-1be2-462f-87db-a27b8c499f33") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:12.389465 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:12.389434    2564 generic.go:356] "Generic (PLEG): container finished" podID="347d9a42-6aa2-4615-b7fd-0ae132d0f512" containerID="bcb98477d1113cf113a68da4ab9118eb99a5ffcc6a720725dd09c956dd36b4ea" exitCode=0
Sep 10 23:02:12.389941 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:12.389521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerDied","Data":"bcb98477d1113cf113a68da4ab9118eb99a5ffcc6a720725dd09c956dd36b4ea"}
Sep 10 23:02:12.391352 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:12.391317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srh7k" event={"ID":"e99ce852-5695-4da2-a352-5d714874db49","Type":"ContainerStarted","Data":"2a29b22495f596ed83e0c143f4bf993bfa92f5b2d5da0d31df81e1c2f462fe7a"}
Sep 10 23:02:13.397495 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:13.397463    2564 generic.go:356] "Generic (PLEG): container finished" podID="347d9a42-6aa2-4615-b7fd-0ae132d0f512" containerID="25c8476ca7324d026154906162f88032f7c136becbcd580c0e9c9b273767b7c0" exitCode=0
Sep 10 23:02:13.398056 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:13.397525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerDied","Data":"25c8476ca7324d026154906162f88032f7c136becbcd580c0e9c9b273767b7c0"}
Sep 10 23:02:13.878863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:13.878820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:13.878863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:13.878867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:13.879053 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:13.878957    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:13.879053 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:13.878961    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:13.879053 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:13.879006    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls podName:b9bed0bc-550e-4da3-9a09-223ff72d236e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:17.878993606 +0000 UTC m=+42.165014785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls") pod "dns-default-8szzn" (UID: "b9bed0bc-550e-4da3-9a09-223ff72d236e") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:13.879053 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:13.879020    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls podName:512332f9-1be2-462f-87db-a27b8c499f33 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:17.879014215 +0000 UTC m=+42.165035395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6685x" (UID: "512332f9-1be2-462f-87db-a27b8c499f33") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:14.403156 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.403079    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" event={"ID":"347d9a42-6aa2-4615-b7fd-0ae132d0f512","Type":"ContainerStarted","Data":"d4fe6eb2fdd84cfe61e32d396a2be52b65ad5c7bfabc1765e066b1de8e498b94"}
Sep 10 23:02:14.404737 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.404695    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srh7k" event={"ID":"e99ce852-5695-4da2-a352-5d714874db49","Type":"ContainerStarted","Data":"5f19bb23fb3ab12e2a6fdf044e5ba0b0e1c7e9b1219f3ac77239cf39d3534619"}
Sep 10 23:02:14.423293 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.423258    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xhqtb" podStartSLOduration=9.227755727 podStartE2EDuration="38.423244978s" totalImagesPullingTime="29.195489251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:14.42210929 +0000 UTC m=+38.708130491" watchObservedRunningTime="2026-09-10 23:02:14.423244978 +0000 UTC m=+38.709266181"
Sep 10 23:02:14.436523 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.436481    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-srh7k" podStartSLOduration=1.766307228 podStartE2EDuration="4.436467715s" totalImagesPullingTime="2.670160487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:14.43608279 +0000 UTC m=+38.722103990" watchObservedRunningTime="2026-09-10 23:02:14.436467715 +0000 UTC m=+38.722488916"
Sep 10 23:02:14.772682 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.772649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"]
Sep 10 23:02:14.793493 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.793469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"]
Sep 10 23:02:14.793613 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.793553    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:14.796067 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.796031    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 23:02:14.796067 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.796045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 23:02:14.796238 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.796056    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gkskh\""
Sep 10 23:02:14.796238 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.796076    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:02:14.885552 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.885528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:14.885662 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.885566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkbg\" (UniqueName: \"kubernetes.io/projected/37c1affc-fda2-48be-b7e9-73ff7344be53-kube-api-access-fqkbg\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:14.986256 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.986230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:14.986364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.986265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkbg\" (UniqueName: \"kubernetes.io/projected/37c1affc-fda2-48be-b7e9-73ff7344be53-kube-api-access-fqkbg\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:14.986409 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:14.986361    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:02:14.986441 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:14.986414    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls podName:37c1affc-fda2-48be-b7e9-73ff7344be53 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:15.486400245 +0000 UTC m=+39.772421452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-grmmr" (UID: "37c1affc-fda2-48be-b7e9-73ff7344be53") : secret "samples-operator-tls" not found
Sep 10 23:02:14.997307 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:14.997158    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkbg\" (UniqueName: \"kubernetes.io/projected/37c1affc-fda2-48be-b7e9-73ff7344be53-kube-api-access-fqkbg\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:15.490867 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.490833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:15.491331 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:15.490963    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:02:15.491331 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:15.491097    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls podName:37c1affc-fda2-48be-b7e9-73ff7344be53 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:16.49107534 +0000 UTC m=+40.777096537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-grmmr" (UID: "37c1affc-fda2-48be-b7e9-73ff7344be53") : secret "samples-operator-tls" not found
Sep 10 23:02:15.756543 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.756480    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"]
Sep 10 23:02:15.776437 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.776413    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"]
Sep 10 23:02:15.776625 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.776492    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.778967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.778874    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:02:15.778967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.778933    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 23:02:15.778967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.778955    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 23:02:15.779191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.778975    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bwsjs\""
Sep 10 23:02:15.779818 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.779802    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 23:02:15.792328 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.792305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01171f17-339c-4d81-a52f-210d94357f06-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.792418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.792401    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnshb\" (UniqueName: \"kubernetes.io/projected/01171f17-339c-4d81-a52f-210d94357f06-kube-api-access-hnshb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.792466 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.792441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01171f17-339c-4d81-a52f-210d94357f06-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.869044 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.869024    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"]
Sep 10 23:02:15.884303 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.884278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.884422 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.884283    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"]
Sep 10 23:02:15.886485 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.886457    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 23:02:15.886575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.886462    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 23:02:15.886575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.886514    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 23:02:15.886575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.886532    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:02:15.886575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.886539    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-drg7j\""
Sep 10 23:02:15.893110 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3c43626-1a61-410e-9ad6-64be44e81554-config\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.893198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpzt\" (UniqueName: \"kubernetes.io/projected/f3c43626-1a61-410e-9ad6-64be44e81554-kube-api-access-jcpzt\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.893198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01171f17-339c-4d81-a52f-210d94357f06-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.893290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c43626-1a61-410e-9ad6-64be44e81554-serving-cert\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.893290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnshb\" (UniqueName: \"kubernetes.io/projected/01171f17-339c-4d81-a52f-210d94357f06-kube-api-access-hnshb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.893348 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01171f17-339c-4d81-a52f-210d94357f06-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.893615 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.893596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01171f17-339c-4d81-a52f-210d94357f06-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.896110 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.896093    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01171f17-339c-4d81-a52f-210d94357f06-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.898430 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.898413    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9p9ht"]
Sep 10 23:02:15.898533 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.898518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"
Sep 10 23:02:15.900918 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.900895    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 23:02:15.901007 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.900952    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mvw8d\""
Sep 10 23:02:15.901007 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.900988    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:02:15.902809 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.902793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnshb\" (UniqueName: \"kubernetes.io/projected/01171f17-339c-4d81-a52f-210d94357f06-kube-api-access-hnshb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5l8bb\" (UID: \"01171f17-339c-4d81-a52f-210d94357f06\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:15.916626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.916610    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"]
Sep 10 23:02:15.916713 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.916635    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"]
Sep 10 23:02:15.916713 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.916649    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9p9ht"]
Sep 10 23:02:15.916829 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.916724    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:15.918934 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.918915    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:02:15.919040 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.918978    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 23:02:15.919040 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.918985    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 23:02:15.919040 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.919014    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5sh4v\""
Sep 10 23:02:15.919194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.919010    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 23:02:15.923605 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.923587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 23:02:15.994122 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxj68\" (UniqueName: \"kubernetes.io/projected/a1de3337-6e3d-4da9-b4a5-6ccf3616bcff-kube-api-access-vxj68\") pod \"volume-data-source-validator-86695b7cc7-8drn6\" (UID: \"a1de3337-6e3d-4da9-b4a5-6ccf3616bcff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"
Sep 10 23:02:15.994212 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zmd\" (UniqueName: \"kubernetes.io/projected/54d614b8-727f-4d5f-8fd0-77390fe3200a-kube-api-access-85zmd\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:15.994212 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c43626-1a61-410e-9ad6-64be44e81554-serving-cert\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.994304 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994230    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54d614b8-727f-4d5f-8fd0-77390fe3200a-serving-cert\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:15.994304 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994251    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54d614b8-727f-4d5f-8fd0-77390fe3200a-trusted-ca\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:15.994304 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3c43626-1a61-410e-9ad6-64be44e81554-config\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.994403 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpzt\" (UniqueName: \"kubernetes.io/projected/f3c43626-1a61-410e-9ad6-64be44e81554-kube-api-access-jcpzt\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.994403 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54d614b8-727f-4d5f-8fd0-77390fe3200a-config\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:15.994766 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.994747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3c43626-1a61-410e-9ad6-64be44e81554-config\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:15.996125 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:15.996110    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3c43626-1a61-410e-9ad6-64be44e81554-serving-cert\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:16.006751 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.006692    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpzt\" (UniqueName: \"kubernetes.io/projected/f3c43626-1a61-410e-9ad6-64be44e81554-kube-api-access-jcpzt\") pod \"service-ca-operator-cf7c776b7-tvkbb\" (UID: \"f3c43626-1a61-410e-9ad6-64be44e81554\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:16.085096 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.085063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"
Sep 10 23:02:16.095497 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.095475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54d614b8-727f-4d5f-8fd0-77390fe3200a-serving-cert\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.095573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.095500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54d614b8-727f-4d5f-8fd0-77390fe3200a-trusted-ca\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.095573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.095540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54d614b8-727f-4d5f-8fd0-77390fe3200a-config\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.095674 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.095574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxj68\" (UniqueName: \"kubernetes.io/projected/a1de3337-6e3d-4da9-b4a5-6ccf3616bcff-kube-api-access-vxj68\") pod \"volume-data-source-validator-86695b7cc7-8drn6\" (UID: \"a1de3337-6e3d-4da9-b4a5-6ccf3616bcff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"
Sep 10 23:02:16.095674 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.095597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zmd\" (UniqueName: \"kubernetes.io/projected/54d614b8-727f-4d5f-8fd0-77390fe3200a-kube-api-access-85zmd\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.096302 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.096281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54d614b8-727f-4d5f-8fd0-77390fe3200a-config\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.096447 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.096432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54d614b8-727f-4d5f-8fd0-77390fe3200a-trusted-ca\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.097572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.097552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54d614b8-727f-4d5f-8fd0-77390fe3200a-serving-cert\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.104166 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.104145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zmd\" (UniqueName: \"kubernetes.io/projected/54d614b8-727f-4d5f-8fd0-77390fe3200a-kube-api-access-85zmd\") pod \"console-operator-696884fb9c-9p9ht\" (UID: \"54d614b8-727f-4d5f-8fd0-77390fe3200a\") " pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.104358 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.104340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxj68\" (UniqueName: \"kubernetes.io/projected/a1de3337-6e3d-4da9-b4a5-6ccf3616bcff-kube-api-access-vxj68\") pod \"volume-data-source-validator-86695b7cc7-8drn6\" (UID: \"a1de3337-6e3d-4da9-b4a5-6ccf3616bcff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"
Sep 10 23:02:16.192899 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.192872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"
Sep 10 23:02:16.211699 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.211680    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"
Sep 10 23:02:16.225496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.225480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:16.385729 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.385690    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb"]
Sep 10 23:02:16.388759 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:16.388738    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01171f17_339c_4d81_a52f_210d94357f06.slice/crio-5a3d4180c00c4f51fe8ec8214db6bc6dcc054d8dc56014c8151d4f638af95b05 WatchSource:0}: Error finding container 5a3d4180c00c4f51fe8ec8214db6bc6dcc054d8dc56014c8151d4f638af95b05: Status 404 returned error can't find the container with id 5a3d4180c00c4f51fe8ec8214db6bc6dcc054d8dc56014c8151d4f638af95b05
Sep 10 23:02:16.410425 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.410398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb" event={"ID":"01171f17-339c-4d81-a52f-210d94357f06","Type":"ContainerStarted","Data":"5a3d4180c00c4f51fe8ec8214db6bc6dcc054d8dc56014c8151d4f638af95b05"}
Sep 10 23:02:16.488632 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.488606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb"]
Sep 10 23:02:16.491246 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:16.491224    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c43626_1a61_410e_9ad6_64be44e81554.slice/crio-835a9f4279091b5354da563616de1e40793aa465f0b838a60c626d0f94e502b9 WatchSource:0}: Error finding container 835a9f4279091b5354da563616de1e40793aa465f0b838a60c626d0f94e502b9: Status 404 returned error can't find the container with id 835a9f4279091b5354da563616de1e40793aa465f0b838a60c626d0f94e502b9
Sep 10 23:02:16.497979 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.497958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:16.498123 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:16.498104    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:02:16.498176 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:16.498159    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls podName:37c1affc-fda2-48be-b7e9-73ff7344be53 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:18.498144694 +0000 UTC m=+42.784165909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-grmmr" (UID: "37c1affc-fda2-48be-b7e9-73ff7344be53") : secret "samples-operator-tls" not found
Sep 10 23:02:16.509509 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.509489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6"]
Sep 10 23:02:16.511798 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:16.511775    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1de3337_6e3d_4da9_b4a5_6ccf3616bcff.slice/crio-f6538bc995b99c1074c6c1a1cc1ea9e092e4c0ff44b773cae2044c0517aff902 WatchSource:0}: Error finding container f6538bc995b99c1074c6c1a1cc1ea9e092e4c0ff44b773cae2044c0517aff902: Status 404 returned error can't find the container with id f6538bc995b99c1074c6c1a1cc1ea9e092e4c0ff44b773cae2044c0517aff902
Sep 10 23:02:16.544329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:16.544277    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9p9ht"]
Sep 10 23:02:16.546820 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:16.546800    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d614b8_727f_4d5f_8fd0_77390fe3200a.slice/crio-dbbb687cbdb7cb9c3ae522979c25bb7c1d3451714c86a4021e268dc0d058d776 WatchSource:0}: Error finding container dbbb687cbdb7cb9c3ae522979c25bb7c1d3451714c86a4021e268dc0d058d776: Status 404 returned error can't find the container with id dbbb687cbdb7cb9c3ae522979c25bb7c1d3451714c86a4021e268dc0d058d776
Sep 10 23:02:17.415818 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:17.415782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" event={"ID":"54d614b8-727f-4d5f-8fd0-77390fe3200a","Type":"ContainerStarted","Data":"dbbb687cbdb7cb9c3ae522979c25bb7c1d3451714c86a4021e268dc0d058d776"}
Sep 10 23:02:17.417406 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:17.417376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6" event={"ID":"a1de3337-6e3d-4da9-b4a5-6ccf3616bcff","Type":"ContainerStarted","Data":"f6538bc995b99c1074c6c1a1cc1ea9e092e4c0ff44b773cae2044c0517aff902"}
Sep 10 23:02:17.419359 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:17.419299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb" event={"ID":"f3c43626-1a61-410e-9ad6-64be44e81554","Type":"ContainerStarted","Data":"835a9f4279091b5354da563616de1e40793aa465f0b838a60c626d0f94e502b9"}
Sep 10 23:02:17.908926 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:17.908879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:17.909681 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:17.909028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:17.909681 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:17.909163    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:17.909681 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:17.909229    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls podName:512332f9-1be2-462f-87db-a27b8c499f33 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:25.909209689 +0000 UTC m=+50.195230887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6685x" (UID: "512332f9-1be2-462f-87db-a27b8c499f33") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:17.909681 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:17.909630    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:17.909681 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:17.909677    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls podName:b9bed0bc-550e-4da3-9a09-223ff72d236e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:25.909662842 +0000 UTC m=+50.195684037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls") pod "dns-default-8szzn" (UID: "b9bed0bc-550e-4da3-9a09-223ff72d236e") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:18.513434 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:18.513397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:18.513626 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:18.513553    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:02:18.513679 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:18.513639    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls podName:37c1affc-fda2-48be-b7e9-73ff7344be53 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:22.513620715 +0000 UTC m=+46.799641909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-grmmr" (UID: "37c1affc-fda2-48be-b7e9-73ff7344be53") : secret "samples-operator-tls" not found
Sep 10 23:02:21.432995 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.432964    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/0.log"
Sep 10 23:02:21.433430 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.433006    2564 generic.go:356] "Generic (PLEG): container finished" podID="54d614b8-727f-4d5f-8fd0-77390fe3200a" containerID="27901ee3b693272d84562913f8257589285a842c56204b4178f1e45938a84727" exitCode=255
Sep 10 23:02:21.433430 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.433087    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" event={"ID":"54d614b8-727f-4d5f-8fd0-77390fe3200a","Type":"ContainerDied","Data":"27901ee3b693272d84562913f8257589285a842c56204b4178f1e45938a84727"}
Sep 10 23:02:21.433430 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.433326    2564 scope.go:117] "RemoveContainer" containerID="27901ee3b693272d84562913f8257589285a842c56204b4178f1e45938a84727"
Sep 10 23:02:21.434936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.434899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb" event={"ID":"01171f17-339c-4d81-a52f-210d94357f06","Type":"ContainerStarted","Data":"10d28a33913b8c9e503118ef8c24311989ae5c480b8a7140a0682102599a7eab"}
Sep 10 23:02:21.436699 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.436659    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6" event={"ID":"a1de3337-6e3d-4da9-b4a5-6ccf3616bcff","Type":"ContainerStarted","Data":"0e29e18ad1599524f43cff566faa8a822106d9df37631007a6a2ade9df019fd3"}
Sep 10 23:02:21.438314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.438286    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb" event={"ID":"f3c43626-1a61-410e-9ad6-64be44e81554","Type":"ContainerStarted","Data":"7e67e8c7aca00976ef9842f6dabdbf5a4f7cb8021de3f112d586ae5420200129"}
Sep 10 23:02:21.464614 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.464575    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb" podStartSLOduration=2.441823547 podStartE2EDuration="6.464561803s" totalImagesPullingTime="4.022738256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:21.463869653 +0000 UTC m=+45.749890854" watchObservedRunningTime="2026-09-10 23:02:21.464561803 +0000 UTC m=+45.750583003"
Sep 10 23:02:21.479971 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.479926    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb" podStartSLOduration=2.559737862 podStartE2EDuration="6.479909153s" totalImagesPullingTime="3.920171291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:21.479152519 +0000 UTC m=+45.765173719" watchObservedRunningTime="2026-09-10 23:02:21.479909153 +0000 UTC m=+45.765930356"
Sep 10 23:02:21.497469 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.497395    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8drn6" podStartSLOduration=2.623528384 podStartE2EDuration="6.497378377s" totalImagesPullingTime="3.873849993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:21.497106023 +0000 UTC m=+45.783127226" watchObservedRunningTime="2026-09-10 23:02:21.497378377 +0000 UTC m=+45.783399583"
Sep 10 23:02:21.794196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.794130    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"]
Sep 10 23:02:21.819450 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.819427    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"]
Sep 10 23:02:21.819578 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.819514    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"
Sep 10 23:02:21.821836 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.821812    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5hxh2\""
Sep 10 23:02:21.837616 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.837591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8knhc\" (UniqueName: \"kubernetes.io/projected/5672bbde-5711-47bf-8c6b-25c40694f37c-kube-api-access-8knhc\") pod \"network-check-source-6f8b768455-hnrpn\" (UID: \"5672bbde-5711-47bf-8c6b-25c40694f37c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"
Sep 10 23:02:21.938765 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.938737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8knhc\" (UniqueName: \"kubernetes.io/projected/5672bbde-5711-47bf-8c6b-25c40694f37c-kube-api-access-8knhc\") pod \"network-check-source-6f8b768455-hnrpn\" (UID: \"5672bbde-5711-47bf-8c6b-25c40694f37c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"
Sep 10 23:02:21.947344 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:21.947323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8knhc\" (UniqueName: \"kubernetes.io/projected/5672bbde-5711-47bf-8c6b-25c40694f37c-kube-api-access-8knhc\") pod \"network-check-source-6f8b768455-hnrpn\" (UID: \"5672bbde-5711-47bf-8c6b-25c40694f37c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"
Sep 10 23:02:22.128415 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.128344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"
Sep 10 23:02:22.413769 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.413682    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn"]
Sep 10 23:02:22.416440 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:22.416413    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5672bbde_5711_47bf_8c6b_25c40694f37c.slice/crio-43b51c899c0ac78042c9ebf2f40371e005f3d9bcf08f146b88e78b5883200251 WatchSource:0}: Error finding container 43b51c899c0ac78042c9ebf2f40371e005f3d9bcf08f146b88e78b5883200251: Status 404 returned error can't find the container with id 43b51c899c0ac78042c9ebf2f40371e005f3d9bcf08f146b88e78b5883200251
Sep 10 23:02:22.446210 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.446178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn" event={"ID":"5672bbde-5711-47bf-8c6b-25c40694f37c","Type":"ContainerStarted","Data":"43b51c899c0ac78042c9ebf2f40371e005f3d9bcf08f146b88e78b5883200251"}
Sep 10 23:02:22.447749 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.447731    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:02:22.448176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.448160    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/0.log"
Sep 10 23:02:22.448269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.448196    2564 generic.go:356] "Generic (PLEG): container finished" podID="54d614b8-727f-4d5f-8fd0-77390fe3200a" containerID="0831d4ac3b42e83bb33ad4c224788e83493fa8d0cc129ba7dee6de1b2628b317" exitCode=255
Sep 10 23:02:22.448269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.448256    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" event={"ID":"54d614b8-727f-4d5f-8fd0-77390fe3200a","Type":"ContainerDied","Data":"0831d4ac3b42e83bb33ad4c224788e83493fa8d0cc129ba7dee6de1b2628b317"}
Sep 10 23:02:22.448374 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.448301    2564 scope.go:117] "RemoveContainer" containerID="27901ee3b693272d84562913f8257589285a842c56204b4178f1e45938a84727"
Sep 10 23:02:22.448572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.448482    2564 scope.go:117] "RemoveContainer" containerID="0831d4ac3b42e83bb33ad4c224788e83493fa8d0cc129ba7dee6de1b2628b317"
Sep 10 23:02:22.448915 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:22.448679    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9p9ht_openshift-console-operator(54d614b8-727f-4d5f-8fd0-77390fe3200a)\"" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" podUID="54d614b8-727f-4d5f-8fd0-77390fe3200a"
Sep 10 23:02:22.543501 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:22.543477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:22.543650 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:22.543627    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 23:02:22.543786 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:22.543770    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls podName:37c1affc-fda2-48be-b7e9-73ff7344be53 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:30.543749639 +0000 UTC m=+54.829770826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-grmmr" (UID: "37c1affc-fda2-48be-b7e9-73ff7344be53") : secret "samples-operator-tls" not found
Sep 10 23:02:23.453336 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:23.453312    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:02:23.453761 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:23.453649    2564 scope.go:117] "RemoveContainer" containerID="0831d4ac3b42e83bb33ad4c224788e83493fa8d0cc129ba7dee6de1b2628b317"
Sep 10 23:02:23.453837 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:23.453817    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9p9ht_openshift-console-operator(54d614b8-727f-4d5f-8fd0-77390fe3200a)\"" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" podUID="54d614b8-727f-4d5f-8fd0-77390fe3200a"
Sep 10 23:02:25.968173 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:25.968140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:25.968173 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:25.968176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:25.968686 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:25.968320    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:25.968686 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:25.968383    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls podName:512332f9-1be2-462f-87db-a27b8c499f33 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:41.968366762 +0000 UTC m=+66.254387945 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6685x" (UID: "512332f9-1be2-462f-87db-a27b8c499f33") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:25.968686 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:25.968437    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:25.968686 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:25.968508    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls podName:b9bed0bc-550e-4da3-9a09-223ff72d236e nodeName:}" failed. No retries permitted until 2026-09-10 23:02:41.968487951 +0000 UTC m=+66.254509134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls") pod "dns-default-8szzn" (UID: "b9bed0bc-550e-4da3-9a09-223ff72d236e") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:26.225697 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:26.225628    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:26.225851 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:26.225698    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:26.225992 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:26.225978    2564 scope.go:117] "RemoveContainer" containerID="0831d4ac3b42e83bb33ad4c224788e83493fa8d0cc129ba7dee6de1b2628b317"
Sep 10 23:02:26.226129 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:26.226110    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9p9ht_openshift-console-operator(54d614b8-727f-4d5f-8fd0-77390fe3200a)\"" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" podUID="54d614b8-727f-4d5f-8fd0-77390fe3200a"
Sep 10 23:02:26.464381 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:26.464350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn" event={"ID":"5672bbde-5711-47bf-8c6b-25c40694f37c","Type":"ContainerStarted","Data":"463e2c51e5d60d2f54d085741905a94a9988fe10b07a7233557bf38a3dd5d259"}
Sep 10 23:02:26.485254 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:26.485165    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hnrpn" podStartSLOduration=1.914454632 podStartE2EDuration="5.485145946s" totalImagesPullingTime="3.570691314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:26.485048351 +0000 UTC m=+50.771069565" watchObservedRunningTime="2026-09-10 23:02:26.485145946 +0000 UTC m=+50.771167147"
Sep 10 23:02:30.603475 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:30.603434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:30.605889 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:30.605868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/37c1affc-fda2-48be-b7e9-73ff7344be53-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-grmmr\" (UID: \"37c1affc-fda2-48be-b7e9-73ff7344be53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:30.701927 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:30.701892    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"
Sep 10 23:02:30.968612 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:30.968581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr"]
Sep 10 23:02:31.481626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:31.481593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr" event={"ID":"37c1affc-fda2-48be-b7e9-73ff7344be53","Type":"ContainerStarted","Data":"e706372e96e290e95274fc4dfd02cd15a60b95defb1046392032088bd4f2897b"}
Sep 10 23:02:33.490036 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:33.490000    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr" event={"ID":"37c1affc-fda2-48be-b7e9-73ff7344be53","Type":"ContainerStarted","Data":"e8e55c01b9003b0db6ad3aa7231a4689d7f52e4d85fddaa7dd01b0d587dfe330"}
Sep 10 23:02:33.490472 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:33.490058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr" event={"ID":"37c1affc-fda2-48be-b7e9-73ff7344be53","Type":"ContainerStarted","Data":"34697640d94e3011f3b635a58a87fbed225a785df018ea7818c043715d7aaa76"}
Sep 10 23:02:33.507957 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:33.507912    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-grmmr" podStartSLOduration=17.631186197 podStartE2EDuration="19.507898058s" totalImagesPullingTime="1.876711861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:33.507241996 +0000 UTC m=+57.793263214" watchObservedRunningTime="2026-09-10 23:02:33.507898058 +0000 UTC m=+57.793919259"
Sep 10 23:02:37.286435 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:37.286391    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dq9mr"
Sep 10 23:02:40.192066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:40.192034    2564 scope.go:117] "RemoveContainer" containerID="0831d4ac3b42e83bb33ad4c224788e83493fa8d0cc129ba7dee6de1b2628b317"
Sep 10 23:02:40.514521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:40.514455    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:02:40.514521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:40.514504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" event={"ID":"54d614b8-727f-4d5f-8fd0-77390fe3200a","Type":"ContainerStarted","Data":"15a5fa831114162bde6cc64ee7d4eb85b8f667290b5f293394323e21ef22c65e"}
Sep 10 23:02:40.514824 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:40.514801    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:40.531413 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:40.531365    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht" podStartSLOduration=21.662069553 podStartE2EDuration="25.531348278s" totalImagesPullingTime="3.869278725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:40.5303342 +0000 UTC m=+64.816355414" watchObservedRunningTime="2026-09-10 23:02:40.531348278 +0000 UTC m=+64.817369481"
Sep 10 23:02:40.968719 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:40.968686    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-9p9ht"
Sep 10 23:02:41.883191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.883161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:41.885577 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.885534    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:02:41.896049 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.896024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae183f5-9da6-46d2-b227-78d5de9e33a5-metrics-certs\") pod \"network-metrics-daemon-9vlkd\" (UID: \"6ae183f5-9da6-46d2-b227-78d5de9e33a5\") " pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:41.984463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.984437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:41.984583 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.984499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:41.984583 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.984526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:41.984583 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.984559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:41.986750 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.986722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/512332f9-1be2-462f-87db-a27b8c499f33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6685x\" (UID: \"512332f9-1be2-462f-87db-a27b8c499f33\") " pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:41.986750 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.986738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhgr\" (UniqueName: \"kubernetes.io/projected/640ae5d2-990e-45ac-b74b-869f5f837a2c-kube-api-access-fbhgr\") pod \"network-check-target-gck4c\" (UID: \"640ae5d2-990e-45ac-b74b-869f5f837a2c\") " pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:41.986877 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.986767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9bed0bc-550e-4da3-9a09-223ff72d236e-metrics-tls\") pod \"dns-default-8szzn\" (UID: \"b9bed0bc-550e-4da3-9a09-223ff72d236e\") " pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:41.987153 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.987139    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:02:41.996751 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:41.996721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b53e2c5-97cd-4de1-a418-40e5fe116a6f-original-pull-secret\") pod \"global-pull-secret-syncer-psv9l\" (UID: \"9b53e2c5-97cd-4de1-a418-40e5fe116a6f\") " pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:42.005736 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.005719    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcwbm\""
Sep 10 23:02:42.014777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.014758    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:42.105553 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.105525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psv9l"
Sep 10 23:02:42.105676 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.105567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-glltp\""
Sep 10 23:02:42.112715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.112685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vlkd"
Sep 10 23:02:42.194037 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.194013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w4w8p\""
Sep 10 23:02:42.202077 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.202037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6685x"
Sep 10 23:02:42.215597 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.215578    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-497ss\""
Sep 10 23:02:42.223618 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.223595    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:42.309221 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.309195    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gck4c"]
Sep 10 23:02:42.312255 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:42.312223    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640ae5d2_990e_45ac_b74b_869f5f837a2c.slice/crio-7ca4869271d631a2c6c85307c6fd03c516278d80d166c4d8cff7e14e075d9722 WatchSource:0}: Error finding container 7ca4869271d631a2c6c85307c6fd03c516278d80d166c4d8cff7e14e075d9722: Status 404 returned error can't find the container with id 7ca4869271d631a2c6c85307c6fd03c516278d80d166c4d8cff7e14e075d9722
Sep 10 23:02:42.390317 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.390284    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psv9l"]
Sep 10 23:02:42.392716 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:42.392684    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b53e2c5_97cd_4de1_a418_40e5fe116a6f.slice/crio-e6e40ea85dc8fe224284314091c793291955bde63560c4fa0eb7a2795595bac5 WatchSource:0}: Error finding container e6e40ea85dc8fe224284314091c793291955bde63560c4fa0eb7a2795595bac5: Status 404 returned error can't find the container with id e6e40ea85dc8fe224284314091c793291955bde63560c4fa0eb7a2795595bac5
Sep 10 23:02:42.411748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.411727    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vlkd"]
Sep 10 23:02:42.414731 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:42.414667    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae183f5_9da6_46d2_b227_78d5de9e33a5.slice/crio-a7d18061c18c6179a139af4b381e31e583acefd1808e80d3dedf073a711a9af2 WatchSource:0}: Error finding container a7d18061c18c6179a139af4b381e31e583acefd1808e80d3dedf073a711a9af2: Status 404 returned error can't find the container with id a7d18061c18c6179a139af4b381e31e583acefd1808e80d3dedf073a711a9af2
Sep 10 23:02:42.481829 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.481807    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6685x"]
Sep 10 23:02:42.483785 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:42.483766    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512332f9_1be2_462f_87db_a27b8c499f33.slice/crio-29e068f0b1dd887f21af43c78914b1bb7e0292c6a64d5e2b0eb614585711dff0 WatchSource:0}: Error finding container 29e068f0b1dd887f21af43c78914b1bb7e0292c6a64d5e2b0eb614585711dff0: Status 404 returned error can't find the container with id 29e068f0b1dd887f21af43c78914b1bb7e0292c6a64d5e2b0eb614585711dff0
Sep 10 23:02:42.505611 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.505591    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8szzn"]
Sep 10 23:02:42.509717 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:42.509681    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bed0bc_550e_4da3_9a09_223ff72d236e.slice/crio-a4614d40082ea4070418e7d0b4ae82cba547b826e704f8d092d51fc0e469193a WatchSource:0}: Error finding container a4614d40082ea4070418e7d0b4ae82cba547b826e704f8d092d51fc0e469193a: Status 404 returned error can't find the container with id a4614d40082ea4070418e7d0b4ae82cba547b826e704f8d092d51fc0e469193a
Sep 10 23:02:42.521357 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.521332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6685x" event={"ID":"512332f9-1be2-462f-87db-a27b8c499f33","Type":"ContainerStarted","Data":"29e068f0b1dd887f21af43c78914b1bb7e0292c6a64d5e2b0eb614585711dff0"}
Sep 10 23:02:42.522501 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.522477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psv9l" event={"ID":"9b53e2c5-97cd-4de1-a418-40e5fe116a6f","Type":"ContainerStarted","Data":"e6e40ea85dc8fe224284314091c793291955bde63560c4fa0eb7a2795595bac5"}
Sep 10 23:02:42.523545 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.523528    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8szzn" event={"ID":"b9bed0bc-550e-4da3-9a09-223ff72d236e","Type":"ContainerStarted","Data":"a4614d40082ea4070418e7d0b4ae82cba547b826e704f8d092d51fc0e469193a"}
Sep 10 23:02:42.525303 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.525280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gck4c" event={"ID":"640ae5d2-990e-45ac-b74b-869f5f837a2c","Type":"ContainerStarted","Data":"ca8bd824389fb554088d2a6037e11d92598eef3e6b03f99495bc97a2495261d7"}
Sep 10 23:02:42.525398 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.525322    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gck4c" event={"ID":"640ae5d2-990e-45ac-b74b-869f5f837a2c","Type":"ContainerStarted","Data":"7ca4869271d631a2c6c85307c6fd03c516278d80d166c4d8cff7e14e075d9722"}
Sep 10 23:02:42.525398 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.525348    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:02:42.526501 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.526482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vlkd" event={"ID":"6ae183f5-9da6-46d2-b227-78d5de9e33a5","Type":"ContainerStarted","Data":"a7d18061c18c6179a139af4b381e31e583acefd1808e80d3dedf073a711a9af2"}
Sep 10 23:02:42.541646 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:42.541607    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gck4c" podStartSLOduration=66.541594942 podStartE2EDuration="1m6.541594942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:42.54116179 +0000 UTC m=+66.827182993" watchObservedRunningTime="2026-09-10 23:02:42.541594942 +0000 UTC m=+66.827616142"
Sep 10 23:02:43.535857 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:43.535764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6685x" event={"ID":"512332f9-1be2-462f-87db-a27b8c499f33","Type":"ContainerStarted","Data":"ab5bb172ea76ead5dfe43753af41d0434f97c6eacaeb9b53243e6e18c609cd93"}
Sep 10 23:02:43.535857 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:43.535830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6685x" event={"ID":"512332f9-1be2-462f-87db-a27b8c499f33","Type":"ContainerStarted","Data":"713be4b45c082e973a1fd3272d93d70d7159acbba3c1c7096cb877db383d8923"}
Sep 10 23:02:44.246147 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.246049    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"]
Sep 10 23:02:44.248753 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.248656    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"]
Sep 10 23:02:44.249096 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.249069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.251933 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.251381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.253224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.252895    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 23:02:44.253224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.252915    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 23:02:44.253224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.253109    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nsxz7\""
Sep 10 23:02:44.253810 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.253643    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 23:02:44.253810 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.253662    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 23:02:44.255021 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.255003    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 23:02:44.262405 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.255294    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 23:02:44.262405 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.255504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 23:02:44.264109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.263679    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 23:02:44.264267 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.264250    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"]
Sep 10 23:02:44.270022 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.267999    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"]
Sep 10 23:02:44.301492 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02c5ea5c-6e68-4078-ab2c-a520933b9d21-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586dd7df64-872w2\" (UID: \"02c5ea5c-6e68-4078-ab2c-a520933b9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.301678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5359c649-55cf-481b-9a4b-41b0aaacc80c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.301678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.301678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgml\" (UniqueName: \"kubernetes.io/projected/02c5ea5c-6e68-4078-ab2c-a520933b9d21-kube-api-access-rtgml\") pod \"managed-serviceaccount-addon-agent-586dd7df64-872w2\" (UID: \"02c5ea5c-6e68-4078-ab2c-a520933b9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.301678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-ca\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.301678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.301678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-hub\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.302016 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.301746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn45\" (UniqueName: \"kubernetes.io/projected/5359c649-55cf-481b-9a4b-41b0aaacc80c-kube-api-access-dpn45\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.402954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn45\" (UniqueName: \"kubernetes.io/projected/5359c649-55cf-481b-9a4b-41b0aaacc80c-kube-api-access-dpn45\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02c5ea5c-6e68-4078-ab2c-a520933b9d21-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586dd7df64-872w2\" (UID: \"02c5ea5c-6e68-4078-ab2c-a520933b9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5359c649-55cf-481b-9a4b-41b0aaacc80c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgml\" (UniqueName: \"kubernetes.io/projected/02c5ea5c-6e68-4078-ab2c-a520933b9d21-kube-api-access-rtgml\") pod \"managed-serviceaccount-addon-agent-586dd7df64-872w2\" (UID: \"02c5ea5c-6e68-4078-ab2c-a520933b9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-ca\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.403241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.403214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-hub\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.404198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.404140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5359c649-55cf-481b-9a4b-41b0aaacc80c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.406583 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.406560    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.407250 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.407224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02c5ea5c-6e68-4078-ab2c-a520933b9d21-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-586dd7df64-872w2\" (UID: \"02c5ea5c-6e68-4078-ab2c-a520933b9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.407352 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.407305    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-hub\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.407925 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.407572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.408718 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.408676    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5359c649-55cf-481b-9a4b-41b0aaacc80c-ca\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.418557 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.418535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn45\" (UniqueName: \"kubernetes.io/projected/5359c649-55cf-481b-9a4b-41b0aaacc80c-kube-api-access-dpn45\") pod \"cluster-proxy-proxy-agent-6c5776bc67-hllxh\" (UID: \"5359c649-55cf-481b-9a4b-41b0aaacc80c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:44.419172 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.419148    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgml\" (UniqueName: \"kubernetes.io/projected/02c5ea5c-6e68-4078-ab2c-a520933b9d21-kube-api-access-rtgml\") pod \"managed-serviceaccount-addon-agent-586dd7df64-872w2\" (UID: \"02c5ea5c-6e68-4078-ab2c-a520933b9d21\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.542278 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.542201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vlkd" event={"ID":"6ae183f5-9da6-46d2-b227-78d5de9e33a5","Type":"ContainerStarted","Data":"5c294d7256a42a21eb85b06891efa689a18bfb484d84de3b881c0f22d8ff2c8b"}
Sep 10 23:02:44.542655 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.542279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vlkd" event={"ID":"6ae183f5-9da6-46d2-b227-78d5de9e33a5","Type":"ContainerStarted","Data":"3f8b17ed5fe09fff0a12eae7c1d33c0a13cf8bb982b61526e12cc49999573228"}
Sep 10 23:02:44.561431 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.561376    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9vlkd" podStartSLOduration=67.354825939 podStartE2EDuration="1m8.561355268s" totalImagesPullingTime="1.206529329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:44.560192706 +0000 UTC m=+68.846213908" watchObservedRunningTime="2026-09-10 23:02:44.561355268 +0000 UTC m=+68.847376467"
Sep 10 23:02:44.583391 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.583366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"
Sep 10 23:02:44.591396 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:44.591370    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"
Sep 10 23:02:45.207952 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.207880    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2"]
Sep 10 23:02:45.211536 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:45.211508    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c5ea5c_6e68_4078_ab2c_a520933b9d21.slice/crio-fdc7aeda94bd8da76b90ba2ff53b9e5814d560d6745d4748b48d240d586d2ffd WatchSource:0}: Error finding container fdc7aeda94bd8da76b90ba2ff53b9e5814d560d6745d4748b48d240d586d2ffd: Status 404 returned error can't find the container with id fdc7aeda94bd8da76b90ba2ff53b9e5814d560d6745d4748b48d240d586d2ffd
Sep 10 23:02:45.218060 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.218034    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh"]
Sep 10 23:02:45.221030 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:45.221003    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5359c649_55cf_481b_9a4b_41b0aaacc80c.slice/crio-e993d62b2257d3789107ef6431761bce634539c62c59949907421dd294345c7c WatchSource:0}: Error finding container e993d62b2257d3789107ef6431761bce634539c62c59949907421dd294345c7c: Status 404 returned error can't find the container with id e993d62b2257d3789107ef6431761bce634539c62c59949907421dd294345c7c
Sep 10 23:02:45.548774 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.548629    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6685x" event={"ID":"512332f9-1be2-462f-87db-a27b8c499f33","Type":"ContainerStarted","Data":"7ef75743f7222229d2b8ef8c4009293e2db7f653a7b7bce14273b86b4e32b67a"}
Sep 10 23:02:45.550136 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.550107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh" event={"ID":"5359c649-55cf-481b-9a4b-41b0aaacc80c","Type":"ContainerStarted","Data":"e993d62b2257d3789107ef6431761bce634539c62c59949907421dd294345c7c"}
Sep 10 23:02:45.552348 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.552263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8szzn" event={"ID":"b9bed0bc-550e-4da3-9a09-223ff72d236e","Type":"ContainerStarted","Data":"eb08afe0f8fb35da4987097c93a12c35a589d280ae4bf40b3c9516a96f586a50"}
Sep 10 23:02:45.552348 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.552311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8szzn" event={"ID":"b9bed0bc-550e-4da3-9a09-223ff72d236e","Type":"ContainerStarted","Data":"a29f5c6b3b66011b42a76f15ddbe9c94082b04d7238cc5ab8e84f740c9efb538"}
Sep 10 23:02:45.552348 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.552340    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:45.553715 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.553667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2" event={"ID":"02c5ea5c-6e68-4078-ab2c-a520933b9d21","Type":"ContainerStarted","Data":"fdc7aeda94bd8da76b90ba2ff53b9e5814d560d6745d4748b48d240d586d2ffd"}
Sep 10 23:02:45.568453 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.568413    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6685x" podStartSLOduration=33.322490873 podStartE2EDuration="35.568399129s" totalImagesPullingTime="2.245908256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:02:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:45.567094601 +0000 UTC m=+69.853115803" watchObservedRunningTime="2026-09-10 23:02:45.568399129 +0000 UTC m=+69.854420329"
Sep 10 23:02:45.584192 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:45.584140    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8szzn" podStartSLOduration=33.159473425 podStartE2EDuration="35.584119304s" totalImagesPullingTime="2.424645879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:45.583171942 +0000 UTC m=+69.869193143" watchObservedRunningTime="2026-09-10 23:02:45.584119304 +0000 UTC m=+69.870140510"
Sep 10 23:02:47.563956 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:47.563639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psv9l" event={"ID":"9b53e2c5-97cd-4de1-a418-40e5fe116a6f","Type":"ContainerStarted","Data":"c2dc982ded23efb7bb0c01c52119cab7b65215c1c4cc79c85d6151ae9c37b385"}
Sep 10 23:02:51.580749 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.580680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2" event={"ID":"02c5ea5c-6e68-4078-ab2c-a520933b9d21","Type":"ContainerStarted","Data":"4926c93345895cd3ff0e948ec58a10b0b3d2086c4639550f35bd33b2127b9efe"}
Sep 10 23:02:51.582916 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.582891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh" event={"ID":"5359c649-55cf-481b-9a4b-41b0aaacc80c","Type":"ContainerStarted","Data":"f04ac50b68dbd228fec732ad51c8493897fc24ad68cd162b9544848e5fe941eb"}
Sep 10 23:02:51.583019 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.582927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh" event={"ID":"5359c649-55cf-481b-9a4b-41b0aaacc80c","Type":"ContainerStarted","Data":"4a7042c484280b4f46cee5f78722840abce6d84a24214639f4a5e2471f274d45"}
Sep 10 23:02:51.583019 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.582940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh" event={"ID":"5359c649-55cf-481b-9a4b-41b0aaacc80c","Type":"ContainerStarted","Data":"8fca18f5fb33df0ab9606e261a66301270cdf174e4189e68c08d6524660b7c5c"}
Sep 10 23:02:51.598361 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.598313    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-psv9l" podStartSLOduration=71.201663557 podStartE2EDuration="1m15.59829976s" totalImagesPullingTime="4.396636203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:47.579611293 +0000 UTC m=+71.865632494" watchObservedRunningTime="2026-09-10 23:02:51.59829976 +0000 UTC m=+75.884320960"
Sep 10 23:02:51.598591 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.598572    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-586dd7df64-872w2" podStartSLOduration=2.021272848 podStartE2EDuration="7.598567124s" totalImagesPullingTime="5.577294276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:51.596659069 +0000 UTC m=+75.882680269" watchObservedRunningTime="2026-09-10 23:02:51.598567124 +0000 UTC m=+75.884588325"
Sep 10 23:02:51.617295 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:51.617242    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c5776bc67-hllxh" podStartSLOduration=2.034209183 podStartE2EDuration="7.617226601s" totalImagesPullingTime="5.583017418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:51.616554206 +0000 UTC m=+75.902575407" watchObservedRunningTime="2026-09-10 23:02:51.617226601 +0000 UTC m=+75.903247803"
Sep 10 23:02:55.559881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.559851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8szzn"
Sep 10 23:02:55.737660 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.737632    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slfmv"]
Sep 10 23:02:55.740184 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.740169    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.742749 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 23:02:55.742878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 23:02:55.742878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 23:02:55.742990 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742854    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 23:02:55.742990 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742866    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 23:02:55.742990 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742947    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 23:02:55.742990 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.742851    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ftsws\""
Sep 10 23:02:55.790805 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-sys\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.790907 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7672bee-4336-4eb2-9369-846e0c36db51-metrics-client-ca\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.790907 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-accelerators-collector-config\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.790983 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.790983 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-textfile\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.790983 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.791068 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.790994    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-root\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.791068 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.791032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-wtmp\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.791068 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.791046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thl9l\" (UniqueName: \"kubernetes.io/projected/d7672bee-4336-4eb2-9369-846e0c36db51-kube-api-access-thl9l\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891680 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-wtmp\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891680 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thl9l\" (UniqueName: \"kubernetes.io/projected/d7672bee-4336-4eb2-9369-846e0c36db51-kube-api-access-thl9l\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891680 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-sys\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7672bee-4336-4eb2-9369-846e0c36db51-metrics-client-ca\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-accelerators-collector-config\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-sys\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-textfile\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-wtmp\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-root\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:55.891897    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:02:55.891930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.891921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7672bee-4336-4eb2-9369-846e0c36db51-root\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.892267 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:55.891954    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls podName:d7672bee-4336-4eb2-9369-846e0c36db51 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:56.391935159 +0000 UTC m=+80.677956341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls") pod "node-exporter-slfmv" (UID: "d7672bee-4336-4eb2-9369-846e0c36db51") : secret "node-exporter-tls" not found
Sep 10 23:02:55.892267 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.892127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-textfile\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.892337 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.892297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-accelerators-collector-config\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.892337 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.892319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7672bee-4336-4eb2-9369-846e0c36db51-metrics-client-ca\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.894158 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.894140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:55.901851 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:55.901830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thl9l\" (UniqueName: \"kubernetes.io/projected/d7672bee-4336-4eb2-9369-846e0c36db51-kube-api-access-thl9l\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:56.396082 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:56.396047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:56.396251 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:56.396156    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:02:56.396251 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:02:56.396208    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls podName:d7672bee-4336-4eb2-9369-846e0c36db51 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:57.396192629 +0000 UTC m=+81.682213808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls") pod "node-exporter-slfmv" (UID: "d7672bee-4336-4eb2-9369-846e0c36db51") : secret "node-exporter-tls" not found
Sep 10 23:02:57.404182 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:57.404151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:57.406492 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:57.406468    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7672bee-4336-4eb2-9369-846e0c36db51-node-exporter-tls\") pod \"node-exporter-slfmv\" (UID: \"d7672bee-4336-4eb2-9369-846e0c36db51\") " pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:57.549141 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:57.549114    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slfmv"
Sep 10 23:02:57.559518 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:02:57.559494    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7672bee_4336_4eb2_9369_846e0c36db51.slice/crio-17a3ee2445e8431c7229615590b76b6abdac90dde9a5c2fe90abff9b2bbe1b9e WatchSource:0}: Error finding container 17a3ee2445e8431c7229615590b76b6abdac90dde9a5c2fe90abff9b2bbe1b9e: Status 404 returned error can't find the container with id 17a3ee2445e8431c7229615590b76b6abdac90dde9a5c2fe90abff9b2bbe1b9e
Sep 10 23:02:57.604808 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:57.604776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slfmv" event={"ID":"d7672bee-4336-4eb2-9369-846e0c36db51","Type":"ContainerStarted","Data":"17a3ee2445e8431c7229615590b76b6abdac90dde9a5c2fe90abff9b2bbe1b9e"}
Sep 10 23:02:59.614488 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:59.614454    2564 generic.go:356] "Generic (PLEG): container finished" podID="d7672bee-4336-4eb2-9369-846e0c36db51" containerID="d5f914dd7176bbc556f8703b59df22b0910bd1bfc1c18c9777035ca8acffab21" exitCode=0
Sep 10 23:02:59.614865 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:02:59.614493    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slfmv" event={"ID":"d7672bee-4336-4eb2-9369-846e0c36db51","Type":"ContainerDied","Data":"d5f914dd7176bbc556f8703b59df22b0910bd1bfc1c18c9777035ca8acffab21"}
Sep 10 23:03:00.620382 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:00.620347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slfmv" event={"ID":"d7672bee-4336-4eb2-9369-846e0c36db51","Type":"ContainerStarted","Data":"2ee8b00b24b348e863c93eb83625266f519f1ac8b07ec5c91d0e3969707be762"}
Sep 10 23:03:00.621011 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:00.620403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slfmv" event={"ID":"d7672bee-4336-4eb2-9369-846e0c36db51","Type":"ContainerStarted","Data":"12e7975f4f9a06fc25739bfb381fd7f6fda6f7f60fb9b7f11e044c40c9f863c2"}
Sep 10 23:03:00.639537 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:00.639489    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slfmv" podStartSLOduration=4.5594013570000005 podStartE2EDuration="5.63947602s" totalImagesPullingTime="1.015964875s" totalInitContainerRuntime="64.109788ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:00.638210933 +0000 UTC m=+84.924232135" watchObservedRunningTime="2026-09-10 23:03:00.63947602 +0000 UTC m=+84.925497221"
Sep 10 23:03:13.538477 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:13.538442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gck4c"
Sep 10 23:03:20.125894 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:20.125865    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srh7k_e99ce852-5695-4da2-a352-5d714874db49/serve-healthcheck-canary/0.log"
Sep 10 23:03:26.718031 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:26.717995    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3c43626-1a61-410e-9ad6-64be44e81554" containerID="7e67e8c7aca00976ef9842f6dabdbf5a4f7cb8021de3f112d586ae5420200129" exitCode=0
Sep 10 23:03:26.718437 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:26.718053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb" event={"ID":"f3c43626-1a61-410e-9ad6-64be44e81554","Type":"ContainerDied","Data":"7e67e8c7aca00976ef9842f6dabdbf5a4f7cb8021de3f112d586ae5420200129"}
Sep 10 23:03:26.718437 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:26.718296    2564 scope.go:117] "RemoveContainer" containerID="7e67e8c7aca00976ef9842f6dabdbf5a4f7cb8021de3f112d586ae5420200129"
Sep 10 23:03:27.724249 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:27.724213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tvkbb" event={"ID":"f3c43626-1a61-410e-9ad6-64be44e81554","Type":"ContainerStarted","Data":"5fb0daead147eb31953e536df4320df677ef077e6448a1a8074f0664f0e7c082"}
Sep 10 23:03:41.786393 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:41.786354    2564 generic.go:356] "Generic (PLEG): container finished" podID="01171f17-339c-4d81-a52f-210d94357f06" containerID="10d28a33913b8c9e503118ef8c24311989ae5c480b8a7140a0682102599a7eab" exitCode=0
Sep 10 23:03:41.786837 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:41.786434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb" event={"ID":"01171f17-339c-4d81-a52f-210d94357f06","Type":"ContainerDied","Data":"10d28a33913b8c9e503118ef8c24311989ae5c480b8a7140a0682102599a7eab"}
Sep 10 23:03:41.786837 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:41.786771    2564 scope.go:117] "RemoveContainer" containerID="10d28a33913b8c9e503118ef8c24311989ae5c480b8a7140a0682102599a7eab"
Sep 10 23:03:42.792386 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:03:42.792349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5l8bb" event={"ID":"01171f17-339c-4d81-a52f-210d94357f06","Type":"ContainerStarted","Data":"c40c48830f8ad0b101565e40f1db78c2b47361187f3521fe958a679663becf4d"}
Sep 10 23:05:29.760378 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.760344    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"]
Sep 10 23:05:29.762385 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.762369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:29.764747 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.764724    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 23:05:29.765473 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.765457    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 23:05:29.765560 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.765487    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gvpvb\""
Sep 10 23:05:29.773029 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.773009    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"]
Sep 10 23:05:29.891197 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.891154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ab64541-f2b8-4a12-a723-9f72f3f7224d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dzqks\" (UID: \"8ab64541-f2b8-4a12-a723-9f72f3f7224d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:29.891324 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.891208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl79c\" (UniqueName: \"kubernetes.io/projected/8ab64541-f2b8-4a12-a723-9f72f3f7224d-kube-api-access-hl79c\") pod \"cert-manager-cainjector-559cdd8c87-dzqks\" (UID: \"8ab64541-f2b8-4a12-a723-9f72f3f7224d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:29.991475 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.991444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ab64541-f2b8-4a12-a723-9f72f3f7224d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dzqks\" (UID: \"8ab64541-f2b8-4a12-a723-9f72f3f7224d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:29.991600 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:29.991476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl79c\" (UniqueName: \"kubernetes.io/projected/8ab64541-f2b8-4a12-a723-9f72f3f7224d-kube-api-access-hl79c\") pod \"cert-manager-cainjector-559cdd8c87-dzqks\" (UID: \"8ab64541-f2b8-4a12-a723-9f72f3f7224d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:30.000193 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:30.000170    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ab64541-f2b8-4a12-a723-9f72f3f7224d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dzqks\" (UID: \"8ab64541-f2b8-4a12-a723-9f72f3f7224d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:30.000346 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:30.000326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl79c\" (UniqueName: \"kubernetes.io/projected/8ab64541-f2b8-4a12-a723-9f72f3f7224d-kube-api-access-hl79c\") pod \"cert-manager-cainjector-559cdd8c87-dzqks\" (UID: \"8ab64541-f2b8-4a12-a723-9f72f3f7224d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:30.070645 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:30.070575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"
Sep 10 23:05:30.337965 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:30.337781    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dzqks"]
Sep 10 23:05:30.340412 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:05:30.340385    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ab64541_f2b8_4a12_a723_9f72f3f7224d.slice/crio-f5953e2e660f21979f34e9b42822a6435ae5a3cfea101e3569a8697d9d432ecb WatchSource:0}: Error finding container f5953e2e660f21979f34e9b42822a6435ae5a3cfea101e3569a8697d9d432ecb: Status 404 returned error can't find the container with id f5953e2e660f21979f34e9b42822a6435ae5a3cfea101e3569a8697d9d432ecb
Sep 10 23:05:31.180991 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:31.180950    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks" event={"ID":"8ab64541-f2b8-4a12-a723-9f72f3f7224d","Type":"ContainerStarted","Data":"f5953e2e660f21979f34e9b42822a6435ae5a3cfea101e3569a8697d9d432ecb"}
Sep 10 23:05:34.199880 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:34.199847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks" event={"ID":"8ab64541-f2b8-4a12-a723-9f72f3f7224d","Type":"ContainerStarted","Data":"8f322efcfacd7978f7abd12dc1700842a7bb38f37d55f92ce65988717f517ceb"}
Sep 10 23:05:34.215282 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:05:34.215239    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dzqks" podStartSLOduration=2.064559685 podStartE2EDuration="5.215225781s" totalImagesPullingTime="3.150666096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:34.214595258 +0000 UTC m=+238.500616460" watchObservedRunningTime="2026-09-10 23:05:34.215225781 +0000 UTC m=+238.501246978"
Sep 10 23:06:04.100537 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.100466    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"]
Sep 10 23:06:04.103741 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.103698    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.107331 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.107313    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:06:04.107690 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.107673    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:06:04.107786 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.107742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z7b9q\""
Sep 10 23:06:04.108144 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.108121    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:06:04.108547 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.108530    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:06:04.127460 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.127430    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"]
Sep 10 23:06:04.208049 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.208015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/660c71df-37a3-4c6f-9de5-2a3d74adec72-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.208213 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.208064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/660c71df-37a3-4c6f-9de5-2a3d74adec72-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.208213 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.208085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/660c71df-37a3-4c6f-9de5-2a3d74adec72-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.208213 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.208140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/660c71df-37a3-4c6f-9de5-2a3d74adec72-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.208213 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.208157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5lz\" (UniqueName: \"kubernetes.io/projected/660c71df-37a3-4c6f-9de5-2a3d74adec72-kube-api-access-hh5lz\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.308965 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.308927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/660c71df-37a3-4c6f-9de5-2a3d74adec72-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.308965 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.308978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/660c71df-37a3-4c6f-9de5-2a3d74adec72-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.309194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.309007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5lz\" (UniqueName: \"kubernetes.io/projected/660c71df-37a3-4c6f-9de5-2a3d74adec72-kube-api-access-hh5lz\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.309194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.309031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/660c71df-37a3-4c6f-9de5-2a3d74adec72-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.309194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.309186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/660c71df-37a3-4c6f-9de5-2a3d74adec72-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.309414 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.309394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/660c71df-37a3-4c6f-9de5-2a3d74adec72-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.311229 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.311199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/660c71df-37a3-4c6f-9de5-2a3d74adec72-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.311351 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.311332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/660c71df-37a3-4c6f-9de5-2a3d74adec72-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.311458 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.311435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/660c71df-37a3-4c6f-9de5-2a3d74adec72-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.318795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.318775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5lz\" (UniqueName: \"kubernetes.io/projected/660c71df-37a3-4c6f-9de5-2a3d74adec72-kube-api-access-hh5lz\") pod \"opendatahub-operator-controller-manager-f5b5b6476-tq5mf\" (UID: \"660c71df-37a3-4c6f-9de5-2a3d74adec72\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.413986 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.413901    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:04.687419 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:04.687390    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"]
Sep 10 23:06:04.690914 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:04.690891    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660c71df_37a3_4c6f_9de5_2a3d74adec72.slice/crio-003371dd9b567cb6ab4433ff070d1f2532497e81c42822bd159608d02112aa5c WatchSource:0}: Error finding container 003371dd9b567cb6ab4433ff070d1f2532497e81c42822bd159608d02112aa5c: Status 404 returned error can't find the container with id 003371dd9b567cb6ab4433ff070d1f2532497e81c42822bd159608d02112aa5c
Sep 10 23:06:05.308651 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:05.308615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf" event={"ID":"660c71df-37a3-4c6f-9de5-2a3d74adec72","Type":"ContainerStarted","Data":"003371dd9b567cb6ab4433ff070d1f2532497e81c42822bd159608d02112aa5c"}
Sep 10 23:06:08.321374 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:08.321336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf" event={"ID":"660c71df-37a3-4c6f-9de5-2a3d74adec72","Type":"ContainerStarted","Data":"dc1c2ed1bd0aeedb0e46a3c179e124bf83442c2bf40cef1046f46c7307af4206"}
Sep 10 23:06:09.326823 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:09.326789    2564 generic.go:356] "Generic (PLEG): container finished" podID="660c71df-37a3-4c6f-9de5-2a3d74adec72" containerID="dc1c2ed1bd0aeedb0e46a3c179e124bf83442c2bf40cef1046f46c7307af4206" exitCode=0
Sep 10 23:06:09.327191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:09.326840    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf" event={"ID":"660c71df-37a3-4c6f-9de5-2a3d74adec72","Type":"ContainerDied","Data":"dc1c2ed1bd0aeedb0e46a3c179e124bf83442c2bf40cef1046f46c7307af4206"}
Sep 10 23:06:10.332277 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:10.332240    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf" event={"ID":"660c71df-37a3-4c6f-9de5-2a3d74adec72","Type":"ContainerStarted","Data":"fc8cf69c8542681333035cc22808b359aa99fa7aa83e4a77fc2a421a241c1f67"}
Sep 10 23:06:10.332776 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:10.332385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:10.354638 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:10.354597    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf" podStartSLOduration=2.160994554 podStartE2EDuration="6.354584117s" totalImagesPullingTime="3.136769144s" totalInitContainerRuntime="1.056820419s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:10.353873862 +0000 UTC m=+274.639895075" watchObservedRunningTime="2026-09-10 23:06:10.354584117 +0000 UTC m=+274.640605318"
Sep 10 23:06:21.338147 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:21.338118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-tq5mf"
Sep 10 23:06:24.201455 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.201424    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"]
Sep 10 23:06:24.209841 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.209820    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.212292 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.212264    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 23:06:24.212688 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.212671    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:06:24.213392 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.213375    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:06:24.213474 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.213391    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 23:06:24.216857 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.216836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"]
Sep 10 23:06:24.345552 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.345523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd56ef4-0e41-4343-ae31-3e7c4f565591-tls-certs\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.345726 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.345642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxp7f\" (UniqueName: \"kubernetes.io/projected/ddd56ef4-0e41-4343-ae31-3e7c4f565591-kube-api-access-kxp7f\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.345797 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.345731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddd56ef4-0e41-4343-ae31-3e7c4f565591-tmp\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.446469 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.446447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxp7f\" (UniqueName: \"kubernetes.io/projected/ddd56ef4-0e41-4343-ae31-3e7c4f565591-kube-api-access-kxp7f\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.446573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.446485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddd56ef4-0e41-4343-ae31-3e7c4f565591-tmp\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.446573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.446518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd56ef4-0e41-4343-ae31-3e7c4f565591-tls-certs\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.448661 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.448634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddd56ef4-0e41-4343-ae31-3e7c4f565591-tmp\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.448802 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.448786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd56ef4-0e41-4343-ae31-3e7c4f565591-tls-certs\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.455343 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.455294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxp7f\" (UniqueName: \"kubernetes.io/projected/ddd56ef4-0e41-4343-ae31-3e7c4f565591-kube-api-access-kxp7f\") pod \"kube-auth-proxy-5d6c69d6b7-xltjl\" (UID: \"ddd56ef4-0e41-4343-ae31-3e7c4f565591\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.520280 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.520250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"
Sep 10 23:06:24.793455 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:24.793384    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl"]
Sep 10 23:06:24.797051 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:24.797025    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd56ef4_0e41_4343_ae31_3e7c4f565591.slice/crio-e8f92c56c4d1f9e0cf45f52092e3d17dec2f445b89ebd6422d1ab6ecbaa18417 WatchSource:0}: Error finding container e8f92c56c4d1f9e0cf45f52092e3d17dec2f445b89ebd6422d1ab6ecbaa18417: Status 404 returned error can't find the container with id e8f92c56c4d1f9e0cf45f52092e3d17dec2f445b89ebd6422d1ab6ecbaa18417
Sep 10 23:06:25.388848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.388814    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl" event={"ID":"ddd56ef4-0e41-4343-ae31-3e7c4f565591","Type":"ContainerStarted","Data":"e8f92c56c4d1f9e0cf45f52092e3d17dec2f445b89ebd6422d1ab6ecbaa18417"}
Sep 10 23:06:25.392296 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.392270    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"]
Sep 10 23:06:25.395369 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.395352    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.397569 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.397550    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 23:06:25.397927 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.397911    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 23:06:25.398211 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.398192    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 23:06:25.398329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.398196    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 23:06:25.398329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.398220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8ktzk\""
Sep 10 23:06:25.407957 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.407938    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"]
Sep 10 23:06:25.554140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkt5\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-kube-api-access-jjkt5\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.554140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.554140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e74768-a6eb-42dc-bda7-603f5476bc96-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.554390 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.554390 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.554390 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.554390 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.554358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655608 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655608 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkt5\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-kube-api-access-jjkt5\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655608 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e74768-a6eb-42dc-bda7-603f5476bc96-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.655886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.655747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.656322 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.656273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.658773 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.658728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e74768-a6eb-42dc-bda7-603f5476bc96-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.659044 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.659020    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.659226 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.659208    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.659296 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.659283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.663203 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.663156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.664431 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.664388    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkt5\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-kube-api-access-jjkt5\") pod \"istiod-openshift-gateway-7b7995d5cd-qzpst\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.705199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.705173    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:25.983431 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:25.983361    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"]
Sep 10 23:06:25.987880 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:25.987854    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e74768_a6eb_42dc_bda7_603f5476bc96.slice/crio-c13c3a89af255ba730ec26ce77c31dc10d8fcf90b7b0f369f317dd062f7baba3 WatchSource:0}: Error finding container c13c3a89af255ba730ec26ce77c31dc10d8fcf90b7b0f369f317dd062f7baba3: Status 404 returned error can't find the container with id c13c3a89af255ba730ec26ce77c31dc10d8fcf90b7b0f369f317dd062f7baba3
Sep 10 23:06:26.394759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:26.394666    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" event={"ID":"e1e74768-a6eb-42dc-bda7-603f5476bc96","Type":"ContainerStarted","Data":"c13c3a89af255ba730ec26ce77c31dc10d8fcf90b7b0f369f317dd062f7baba3"}
Sep 10 23:06:28.407834 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:28.407792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl" event={"ID":"ddd56ef4-0e41-4343-ae31-3e7c4f565591","Type":"ContainerStarted","Data":"afb692a8f9207f5e4eb118347281a503dff7c4c0a44fbe13051fa6e1f3027cff"}
Sep 10 23:06:28.424446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:28.424270    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-xltjl" podStartSLOduration=1.702000787 podStartE2EDuration="4.424251382s" totalImagesPullingTime="2.722250595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:28.423941983 +0000 UTC m=+292.709963187" watchObservedRunningTime="2026-09-10 23:06:28.424251382 +0000 UTC m=+292.710272587"
Sep 10 23:06:29.298345 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:29.298312    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:06:29.298411 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:29.298379    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:06:29.414131 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:29.414092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" event={"ID":"e1e74768-a6eb-42dc-bda7-603f5476bc96","Type":"ContainerStarted","Data":"80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905"}
Sep 10 23:06:29.435675 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:29.435620    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" podStartSLOduration=1.12777675 podStartE2EDuration="4.435604471s" totalImagesPullingTime="3.307827721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:29.434287933 +0000 UTC m=+293.720309137" watchObservedRunningTime="2026-09-10 23:06:29.435604471 +0000 UTC m=+293.721625671"
Sep 10 23:06:30.418933 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:30.418895    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:30.421098 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:30.420733    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qzpst container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 23:06:30.421098 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:30.420792    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" podUID="e1e74768-a6eb-42dc-bda7-603f5476bc96" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 23:06:31.424104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:31.424075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:34.864231 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:34.864200    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rjxmt"]
Sep 10 23:06:34.867283 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:34.867266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:34.869713 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:34.869680    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 23:06:34.869821 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:34.869727    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-g2v7c\""
Sep 10 23:06:34.875609 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:34.875577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rjxmt"]
Sep 10 23:06:35.028404 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.028375    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95903eff-18ad-4299-bf87-4dc5f442e7b0-cert\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.028551 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.028433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc8p\" (UniqueName: \"kubernetes.io/projected/95903eff-18ad-4299-bf87-4dc5f442e7b0-kube-api-access-7rc8p\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.129913 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.129841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95903eff-18ad-4299-bf87-4dc5f442e7b0-cert\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.130035 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:35.129854    2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 23:06:35.130035 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.129932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc8p\" (UniqueName: \"kubernetes.io/projected/95903eff-18ad-4299-bf87-4dc5f442e7b0-kube-api-access-7rc8p\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.130035 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:35.130005    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95903eff-18ad-4299-bf87-4dc5f442e7b0-cert podName:95903eff-18ad-4299-bf87-4dc5f442e7b0 nodeName:}" failed. No retries permitted until 2026-09-10 23:06:35.629989857 +0000 UTC m=+299.916011037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/95903eff-18ad-4299-bf87-4dc5f442e7b0-cert") pod "odh-model-controller-7998c67dc4-rjxmt" (UID: "95903eff-18ad-4299-bf87-4dc5f442e7b0") : secret "odh-model-controller-webhook-cert" not found
Sep 10 23:06:35.138302 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.138273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc8p\" (UniqueName: \"kubernetes.io/projected/95903eff-18ad-4299-bf87-4dc5f442e7b0-kube-api-access-7rc8p\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.492006 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.491977    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qpfwp"]
Sep 10 23:06:35.494808 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.494791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.496911 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.496890    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 23:06:35.497035 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.496912    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-7zkg7\""
Sep 10 23:06:35.506359 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.506337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qpfwp"]
Sep 10 23:06:35.638775 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.638749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95903eff-18ad-4299-bf87-4dc5f442e7b0-cert\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.638922 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.638796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2r4h\" (UniqueName: \"kubernetes.io/projected/031f92c5-065e-48cf-8919-5064c114610d-kube-api-access-x2r4h\") pod \"model-serving-api-54d89bd5fc-qpfwp\" (UID: \"031f92c5-065e-48cf-8919-5064c114610d\") " pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.638922 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.638822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031f92c5-065e-48cf-8919-5064c114610d-tls-certs\") pod \"model-serving-api-54d89bd5fc-qpfwp\" (UID: \"031f92c5-065e-48cf-8919-5064c114610d\") " pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.641042 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.641022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95903eff-18ad-4299-bf87-4dc5f442e7b0-cert\") pod \"odh-model-controller-7998c67dc4-rjxmt\" (UID: \"95903eff-18ad-4299-bf87-4dc5f442e7b0\") " pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.739815 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.739783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2r4h\" (UniqueName: \"kubernetes.io/projected/031f92c5-065e-48cf-8919-5064c114610d-kube-api-access-x2r4h\") pod \"model-serving-api-54d89bd5fc-qpfwp\" (UID: \"031f92c5-065e-48cf-8919-5064c114610d\") " pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.739916 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.739823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031f92c5-065e-48cf-8919-5064c114610d-tls-certs\") pod \"model-serving-api-54d89bd5fc-qpfwp\" (UID: \"031f92c5-065e-48cf-8919-5064c114610d\") " pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.742178 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.742134    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031f92c5-065e-48cf-8919-5064c114610d-tls-certs\") pod \"model-serving-api-54d89bd5fc-qpfwp\" (UID: \"031f92c5-065e-48cf-8919-5064c114610d\") " pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.748962 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.748942    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2r4h\" (UniqueName: \"kubernetes.io/projected/031f92c5-065e-48cf-8919-5064c114610d-kube-api-access-x2r4h\") pod \"model-serving-api-54d89bd5fc-qpfwp\" (UID: \"031f92c5-065e-48cf-8919-5064c114610d\") " pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:35.777757 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.777737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:35.805309 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:35.805289    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:36.050728 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.050692    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rjxmt"]
Sep 10 23:06:36.053203 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:36.053175    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95903eff_18ad_4299_bf87_4dc5f442e7b0.slice/crio-4faecd47075d366a6eb27d43f3b458a16477da955e7c74e815a11143c4be2e22 WatchSource:0}: Error finding container 4faecd47075d366a6eb27d43f3b458a16477da955e7c74e815a11143c4be2e22: Status 404 returned error can't find the container with id 4faecd47075d366a6eb27d43f3b458a16477da955e7c74e815a11143c4be2e22
Sep 10 23:06:36.079325 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.079293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qpfwp"]
Sep 10 23:06:36.081482 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:36.081445    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031f92c5_065e_48cf_8919_5064c114610d.slice/crio-ba42be77dc725c0416bfb257662e96d2b3532eff863c51900fdb76cfe749fec2 WatchSource:0}: Error finding container ba42be77dc725c0416bfb257662e96d2b3532eff863c51900fdb76cfe749fec2: Status 404 returned error can't find the container with id ba42be77dc725c0416bfb257662e96d2b3532eff863c51900fdb76cfe749fec2
Sep 10 23:06:36.088777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.088757    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:06:36.091423 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.091402    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:06:36.103195 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.103177    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:06:36.444542 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.444510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp" event={"ID":"031f92c5-065e-48cf-8919-5064c114610d","Type":"ContainerStarted","Data":"ba42be77dc725c0416bfb257662e96d2b3532eff863c51900fdb76cfe749fec2"}
Sep 10 23:06:36.446791 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:36.445607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt" event={"ID":"95903eff-18ad-4299-bf87-4dc5f442e7b0","Type":"ContainerStarted","Data":"4faecd47075d366a6eb27d43f3b458a16477da955e7c74e815a11143c4be2e22"}
Sep 10 23:06:39.461961 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:39.461927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt" event={"ID":"95903eff-18ad-4299-bf87-4dc5f442e7b0","Type":"ContainerStarted","Data":"9370043cb1269883f61c149e260dabde3130fae647d06d88b0ec581ee0c8f647"}
Sep 10 23:06:39.462373 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:39.462012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:39.463496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:39.463451    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp" event={"ID":"031f92c5-065e-48cf-8919-5064c114610d","Type":"ContainerStarted","Data":"17dba71fe61029f1f611340f1864b49751508cc71e94d1d5c6609880d046cf10"}
Sep 10 23:06:39.463618 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:39.463538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:06:39.479914 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:39.479871    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt" podStartSLOduration=2.559831415 podStartE2EDuration="5.47985689s" totalImagesPullingTime="2.920025475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:39.478479852 +0000 UTC m=+303.764501052" watchObservedRunningTime="2026-09-10 23:06:39.47985689 +0000 UTC m=+303.765878091"
Sep 10 23:06:39.498576 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:39.498532    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp" podStartSLOduration=1.6541655259999999 podStartE2EDuration="4.498519889s" totalImagesPullingTime="2.844354363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:39.498109081 +0000 UTC m=+303.784130293" watchObservedRunningTime="2026-09-10 23:06:39.498519889 +0000 UTC m=+303.784541090"
Sep 10 23:06:41.728117 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.728082    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"]
Sep 10 23:06:41.735121 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.735096    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.751344 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.751319    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"]
Sep 10 23:06:41.782764 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zm48\" (UniqueName: \"kubernetes.io/projected/380c33da-7499-41ca-8b55-b4251aa997ba-kube-api-access-9zm48\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.782883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/380c33da-7499-41ca-8b55-b4251aa997ba-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.782883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.782883 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380c33da-7499-41ca-8b55-b4251aa997ba-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.783046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/380c33da-7499-41ca-8b55-b4251aa997ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.783046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782958    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.783046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.782995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884261 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884234    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/380c33da-7499-41ca-8b55-b4251aa997ba-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380c33da-7499-41ca-8b55-b4251aa997ba-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/380c33da-7499-41ca-8b55-b4251aa997ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.884674 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.884445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zm48\" (UniqueName: \"kubernetes.io/projected/380c33da-7499-41ca-8b55-b4251aa997ba-kube-api-access-9zm48\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.885211 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.885188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/380c33da-7499-41ca-8b55-b4251aa997ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.887198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.887172    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.887276 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.887195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/380c33da-7499-41ca-8b55-b4251aa997ba-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.887276 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.887198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.887276 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.887264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/380c33da-7499-41ca-8b55-b4251aa997ba-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.899518 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.899495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380c33da-7499-41ca-8b55-b4251aa997ba-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:41.899813 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:41.899794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zm48\" (UniqueName: \"kubernetes.io/projected/380c33da-7499-41ca-8b55-b4251aa997ba-kube-api-access-9zm48\") pod \"istiod-openshift-gateway-5677ccb49b-dh2sn\" (UID: \"380c33da-7499-41ca-8b55-b4251aa997ba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:42.044284 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.044225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:42.326969 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.326943    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"]
Sep 10 23:06:42.328534 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:42.328511    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380c33da_7499_41ca_8b55_b4251aa997ba.slice/crio-bec90429c80b98081adabd9a12984fabbb224ad85eb2afc60c9fcfc1298aa314 WatchSource:0}: Error finding container bec90429c80b98081adabd9a12984fabbb224ad85eb2afc60c9fcfc1298aa314: Status 404 returned error can't find the container with id bec90429c80b98081adabd9a12984fabbb224ad85eb2afc60c9fcfc1298aa314
Sep 10 23:06:42.330742 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.330696    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:06:42.330823 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.330770    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:06:42.477771 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.477737    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn" event={"ID":"380c33da-7499-41ca-8b55-b4251aa997ba","Type":"ContainerStarted","Data":"b11386b9f80d3de68b7d24446e73cf5965f36a8b1ecc93826aacf05673ebeb31"}
Sep 10 23:06:42.477938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.477785    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn" event={"ID":"380c33da-7499-41ca-8b55-b4251aa997ba","Type":"ContainerStarted","Data":"bec90429c80b98081adabd9a12984fabbb224ad85eb2afc60c9fcfc1298aa314"}
Sep 10 23:06:42.477938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.477864    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:42.514104 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:42.514061    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn" podStartSLOduration=1.5140475279999999 podStartE2EDuration="1.514047528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:42.513562477 +0000 UTC m=+306.799583681" watchObservedRunningTime="2026-09-10 23:06:42.514047528 +0000 UTC m=+306.800068728"
Sep 10 23:06:43.487613 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.487576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dh2sn"
Sep 10 23:06:43.627529 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.627499    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"]
Sep 10 23:06:43.627734 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.627700    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" podUID="e1e74768-a6eb-42dc-bda7-603f5476bc96" containerName="discovery" containerID="cri-o://80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905" gracePeriod=30
Sep 10 23:06:43.862391 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.862369    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:43.899673 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899650    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-ca-configmap\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.899781 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899677    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-kubeconfig\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.899781 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-dns-cert\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.899868 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899849    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e74768-a6eb-42dc-bda7-603f5476bc96-local-certs\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.899915 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899885    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjkt5\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-kube-api-access-jjkt5\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.899968 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899914    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-token\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.899968 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.899944    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-cacerts\") pod \"e1e74768-a6eb-42dc-bda7-603f5476bc96\" (UID: \"e1e74768-a6eb-42dc-bda7-603f5476bc96\") "
Sep 10 23:06:43.900204 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.900178    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:06:43.902086 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.902063    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:06:43.902222 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.902170    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-cacerts" (OuterVolumeSpecName: "cacerts") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:06:43.902290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.902255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:06:43.902384 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.902364    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e74768-a6eb-42dc-bda7-603f5476bc96-local-certs" (OuterVolumeSpecName: "local-certs") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:06:43.902431 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.902385    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-kube-api-access-jjkt5" (OuterVolumeSpecName: "kube-api-access-jjkt5") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "kube-api-access-jjkt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:06:43.902522 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:43.902502    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1e74768-a6eb-42dc-bda7-603f5476bc96" (UID: "e1e74768-a6eb-42dc-bda7-603f5476bc96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:06:44.001446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001389    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-ca-configmap\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.001446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001414    2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-kubeconfig\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.001446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001424    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-csr-dns-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.001446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001432    2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1e74768-a6eb-42dc-bda7-603f5476bc96-local-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.001446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001441    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjkt5\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-kube-api-access-jjkt5\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.001446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001450    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1e74768-a6eb-42dc-bda7-603f5476bc96-istio-token\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.001805 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.001458    2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e1e74768-a6eb-42dc-bda7-603f5476bc96-cacerts\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.366229 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.366152    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"]
Sep 10 23:06:44.366513 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.366496    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e74768-a6eb-42dc-bda7-603f5476bc96" containerName="discovery"
Sep 10 23:06:44.366584 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.366517    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e74768-a6eb-42dc-bda7-603f5476bc96" containerName="discovery"
Sep 10 23:06:44.366638 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.366627    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e74768-a6eb-42dc-bda7-603f5476bc96" containerName="discovery"
Sep 10 23:06:44.368694 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.368676    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:44.371254 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.371224    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 23:06:44.371254 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.371250    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4sq8b\""
Sep 10 23:06:44.379243 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.379220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"]
Sep 10 23:06:44.403828 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.403798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac2848c-abed-4957-a0f7-80b53ca22e77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:44.403913 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.403837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnxb\" (UniqueName: \"kubernetes.io/projected/3ac2848c-abed-4957-a0f7-80b53ca22e77-kube-api-access-6mnxb\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:44.489333 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.489305    2564 generic.go:356] "Generic (PLEG): container finished" podID="e1e74768-a6eb-42dc-bda7-603f5476bc96" containerID="80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905" exitCode=0
Sep 10 23:06:44.489690 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.489349    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"
Sep 10 23:06:44.489690 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.489362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" event={"ID":"e1e74768-a6eb-42dc-bda7-603f5476bc96","Type":"ContainerDied","Data":"80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905"}
Sep 10 23:06:44.489690 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.489412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst" event={"ID":"e1e74768-a6eb-42dc-bda7-603f5476bc96","Type":"ContainerDied","Data":"c13c3a89af255ba730ec26ce77c31dc10d8fcf90b7b0f369f317dd062f7baba3"}
Sep 10 23:06:44.489690 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.489432    2564 scope.go:117] "RemoveContainer" containerID="80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905"
Sep 10 23:06:44.497694 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.497676    2564 scope.go:117] "RemoveContainer" containerID="80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905"
Sep 10 23:06:44.498021 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:44.498003    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905\": container with ID starting with 80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905 not found: ID does not exist" containerID="80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905"
Sep 10 23:06:44.498065 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.498037    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905"} err="failed to get container status \"80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905\": rpc error: code = NotFound desc = could not find container \"80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905\": container with ID starting with 80168263bdc57fd7a28fac53f54516a0d7f332fe5ba0aa12068b9b76f3cdd905 not found: ID does not exist"
Sep 10 23:06:44.504548 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.504525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac2848c-abed-4957-a0f7-80b53ca22e77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:44.504651 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.504556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnxb\" (UniqueName: \"kubernetes.io/projected/3ac2848c-abed-4957-a0f7-80b53ca22e77-kube-api-access-6mnxb\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:44.504732 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:44.504646    2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 10 23:06:44.504732 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:44.504716    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac2848c-abed-4957-a0f7-80b53ca22e77-cert podName:3ac2848c-abed-4957-a0f7-80b53ca22e77 nodeName:}" failed. No retries permitted until 2026-09-10 23:06:45.004683669 +0000 UTC m=+309.290704848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3ac2848c-abed-4957-a0f7-80b53ca22e77-cert") pod "llmisvc-controller-manager-777c6bd7d5-d4qt8" (UID: "3ac2848c-abed-4957-a0f7-80b53ca22e77") : secret "llmisvc-webhook-server-cert" not found
Sep 10 23:06:44.512867 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.512848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnxb\" (UniqueName: \"kubernetes.io/projected/3ac2848c-abed-4957-a0f7-80b53ca22e77-kube-api-access-6mnxb\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:44.514923 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.514900    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"]
Sep 10 23:06:44.521918 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:44.521894    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qzpst"]
Sep 10 23:06:45.009125 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:45.009096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac2848c-abed-4957-a0f7-80b53ca22e77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:45.011456 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:45.011433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac2848c-abed-4957-a0f7-80b53ca22e77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d4qt8\" (UID: \"3ac2848c-abed-4957-a0f7-80b53ca22e77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:45.278960 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:45.278884    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:06:45.555455 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:45.555384    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"]
Sep 10 23:06:45.557741 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:06:45.557714    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3ac2848c_abed_4957_a0f7_80b53ca22e77.slice/crio-01dde6691add79bcdbcd8b4882ea8d95414b0de1d0e3fb6057bf2bebc61c7fd4 WatchSource:0}: Error finding container 01dde6691add79bcdbcd8b4882ea8d95414b0de1d0e3fb6057bf2bebc61c7fd4: Status 404 returned error can't find the container with id 01dde6691add79bcdbcd8b4882ea8d95414b0de1d0e3fb6057bf2bebc61c7fd4
Sep 10 23:06:45.559555 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:45.559531    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:06:46.205572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:46.201436    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e74768-a6eb-42dc-bda7-603f5476bc96" path="/var/lib/kubelet/pods/e1e74768-a6eb-42dc-bda7-603f5476bc96/volumes"
Sep 10 23:06:46.500335 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:46.500266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8" event={"ID":"3ac2848c-abed-4957-a0f7-80b53ca22e77","Type":"ContainerStarted","Data":"01dde6691add79bcdbcd8b4882ea8d95414b0de1d0e3fb6057bf2bebc61c7fd4"}
Sep 10 23:06:48.010407 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:48.010367    2564 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:02c91f6b395a6b06d17b8985a2db90aef7b09feedff77eff1f7c269161263a9b: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2"
Sep 10 23:06:48.010770 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:48.010482    2564 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod llmisvc-controller-manager-777c6bd7d5-d4qt8_opendatahub(3ac2848c-abed-4957-a0f7-80b53ca22e77): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:02c91f6b395a6b06d17b8985a2db90aef7b09feedff77eff1f7c269161263a9b: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError"
Sep 10 23:06:48.010770 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:48.010513    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:02c91f6b395a6b06d17b8985a2db90aef7b09feedff77eff1f7c269161263a9b: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8" podUID="3ac2848c-abed-4957-a0f7-80b53ca22e77"
Sep 10 23:06:48.508676 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:06:48.508636    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:02c91f6b395a6b06d17b8985a2db90aef7b09feedff77eff1f7c269161263a9b: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8" podUID="3ac2848c-abed-4957-a0f7-80b53ca22e77"
Sep 10 23:06:50.470559 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:50.470525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rjxmt"
Sep 10 23:06:50.472443 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:06:50.472417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qpfwp"
Sep 10 23:07:03.568530 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:07:03.568447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8" event={"ID":"3ac2848c-abed-4957-a0f7-80b53ca22e77","Type":"ContainerStarted","Data":"ea6e4bfe8b1a85aa99aa5e0c3271dd34665f42e0fc42cff9eb31c45c75b8dae4"}
Sep 10 23:07:03.568925 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:07:03.568646    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:07:03.586040 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:07:03.585995    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8" podStartSLOduration=1.9170364850000001 podStartE2EDuration="19.585982138s" totalImagesPullingTime="17.668945653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-10 23:07:03.58530917 +0000 UTC m=+327.871330373" watchObservedRunningTime="2026-09-10 23:07:03.585982138 +0000 UTC m=+327.872003339"
Sep 10 23:07:34.575468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:07:34.575397    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d4qt8"
Sep 10 23:08:13.316781 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.316746    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4mqqp"]
Sep 10 23:08:13.319949 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.319932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:13.322621 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.322600    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bftv6\""
Sep 10 23:08:13.323214 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.323197    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:08:13.323214 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.323208    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:08:13.335369 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.335346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrjk\" (UniqueName: \"kubernetes.io/projected/bf51295c-0259-4d53-95f3-2d856081fbe8-kube-api-access-vxrjk\") pod \"authorino-operator-76b478bd78-4mqqp\" (UID: \"bf51295c-0259-4d53-95f3-2d856081fbe8\") " pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:13.337125 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.337105    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4mqqp"]
Sep 10 23:08:13.436493 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.436466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrjk\" (UniqueName: \"kubernetes.io/projected/bf51295c-0259-4d53-95f3-2d856081fbe8-kube-api-access-vxrjk\") pod \"authorino-operator-76b478bd78-4mqqp\" (UID: \"bf51295c-0259-4d53-95f3-2d856081fbe8\") " pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:13.448140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.448103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrjk\" (UniqueName: \"kubernetes.io/projected/bf51295c-0259-4d53-95f3-2d856081fbe8-kube-api-access-vxrjk\") pod \"authorino-operator-76b478bd78-4mqqp\" (UID: \"bf51295c-0259-4d53-95f3-2d856081fbe8\") " pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:13.629688 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.629626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:13.925525 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:13.925457    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4mqqp"]
Sep 10 23:08:13.928167 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:08:13.928142    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf51295c_0259_4d53_95f3_2d856081fbe8.slice/crio-cac4558446415c6dbf56e8dae430a967add17cfed52847b93ff6069824dccdac WatchSource:0}: Error finding container cac4558446415c6dbf56e8dae430a967add17cfed52847b93ff6069824dccdac: Status 404 returned error can't find the container with id cac4558446415c6dbf56e8dae430a967add17cfed52847b93ff6069824dccdac
Sep 10 23:08:14.875841 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:14.875763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp" event={"ID":"bf51295c-0259-4d53-95f3-2d856081fbe8","Type":"ContainerStarted","Data":"cac4558446415c6dbf56e8dae430a967add17cfed52847b93ff6069824dccdac"}
Sep 10 23:08:16.886067 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:16.886031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp" event={"ID":"bf51295c-0259-4d53-95f3-2d856081fbe8","Type":"ContainerStarted","Data":"36822d1e96dc8a658155035cb87259bbf6894ce2678b514a1ffea11d43945bc2"}
Sep 10 23:08:16.886464 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:16.886153    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:16.906155 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:16.906093    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp" podStartSLOduration=1.6883797820000002 podStartE2EDuration="3.906072764s" totalImagesPullingTime="2.217692982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:16.904209052 +0000 UTC m=+401.190230255" watchObservedRunningTime="2026-09-10 23:08:16.906072764 +0000 UTC m=+401.192093984"
Sep 10 23:08:25.323288 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.323251    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"]
Sep 10 23:08:25.326560 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.326544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:25.333458 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.333433    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2b289\""
Sep 10 23:08:25.333571 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.333433    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 23:08:25.340823 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.340800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"]
Sep 10 23:08:25.422943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.422918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgckr\" (UniqueName: \"kubernetes.io/projected/854df844-debd-457d-911c-672e87cdb96c-kube-api-access-tgckr\") pod \"dns-operator-controller-manager-7b84cd7c9c-jfmtk\" (UID: \"854df844-debd-457d-911c-672e87cdb96c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:25.523748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.523720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgckr\" (UniqueName: \"kubernetes.io/projected/854df844-debd-457d-911c-672e87cdb96c-kube-api-access-tgckr\") pod \"dns-operator-controller-manager-7b84cd7c9c-jfmtk\" (UID: \"854df844-debd-457d-911c-672e87cdb96c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:25.548210 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.548179    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgckr\" (UniqueName: \"kubernetes.io/projected/854df844-debd-457d-911c-672e87cdb96c-kube-api-access-tgckr\") pod \"dns-operator-controller-manager-7b84cd7c9c-jfmtk\" (UID: \"854df844-debd-457d-911c-672e87cdb96c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:25.635956 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.635899    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:25.910380 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.910352    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"]
Sep 10 23:08:25.911969 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:08:25.911944    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854df844_debd_457d_911c_672e87cdb96c.slice/crio-ab06ce31258cf0eecc50f67a11a3261cb7e5a428348d2a887561a8571a8f96d4 WatchSource:0}: Error finding container ab06ce31258cf0eecc50f67a11a3261cb7e5a428348d2a887561a8571a8f96d4: Status 404 returned error can't find the container with id ab06ce31258cf0eecc50f67a11a3261cb7e5a428348d2a887561a8571a8f96d4
Sep 10 23:08:25.925611 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:25.925587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk" event={"ID":"854df844-debd-457d-911c-672e87cdb96c","Type":"ContainerStarted","Data":"ab06ce31258cf0eecc50f67a11a3261cb7e5a428348d2a887561a8571a8f96d4"}
Sep 10 23:08:27.892482 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:27.892453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-4mqqp"
Sep 10 23:08:28.943395 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:28.943356    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk" event={"ID":"854df844-debd-457d-911c-672e87cdb96c","Type":"ContainerStarted","Data":"b10868583252927f7cf9491e69919fbfdfdd17866cf572381f466f078ed36f60"}
Sep 10 23:08:28.943808 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:28.943496    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:28.967322 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:28.967268    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk" podStartSLOduration=1.933449016 podStartE2EDuration="3.967252809s" totalImagesPullingTime="2.033803793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:28.965095168 +0000 UTC m=+413.251116401" watchObservedRunningTime="2026-09-10 23:08:28.967252809 +0000 UTC m=+413.253274010"
Sep 10 23:08:39.949756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:39.949722    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jfmtk"
Sep 10 23:08:40.568203 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.568153    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"]
Sep 10 23:08:40.572722 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.572684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.575269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.575247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-thk95\""
Sep 10 23:08:40.587827 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.587802    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"]
Sep 10 23:08:40.638817 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.638797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnnn\" (UniqueName: \"kubernetes.io/projected/37593210-7b63-459e-92cb-dd578993c392-kube-api-access-flnnn\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.638924 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.638844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37593210-7b63-459e-92cb-dd578993c392-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.739451 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.739423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnnn\" (UniqueName: \"kubernetes.io/projected/37593210-7b63-459e-92cb-dd578993c392-kube-api-access-flnnn\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.739549 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.739473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37593210-7b63-459e-92cb-dd578993c392-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.739896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.739878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37593210-7b63-459e-92cb-dd578993c392-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.754094 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.754068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnnn\" (UniqueName: \"kubernetes.io/projected/37593210-7b63-459e-92cb-dd578993c392-kube-api-access-flnnn\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.882904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.882849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:40.921882 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.921844    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"]
Sep 10 23:08:40.931202 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.931175    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"]
Sep 10 23:08:40.960721 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.960684    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"]
Sep 10 23:08:40.964561 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.964543    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:40.982607 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:40.982582    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"]
Sep 10 23:08:41.042619 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.042590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hkp\" (UniqueName: \"kubernetes.io/projected/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-kube-api-access-p9hkp\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-tn68p\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.042779 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.042626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-tn68p\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.143066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.142995    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hkp\" (UniqueName: \"kubernetes.io/projected/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-kube-api-access-p9hkp\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-tn68p\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.143066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.143030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-tn68p\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.143350 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.143334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-tn68p\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.152842 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.152819    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hkp\" (UniqueName: \"kubernetes.io/projected/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-kube-api-access-p9hkp\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-tn68p\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.276093 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.276071    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:41.575531 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:41.575505    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"]
Sep 10 23:08:41.577174 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:08:41.577147    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0435be0_4cb5_4345_b9fc_1258d5c5e7cb.slice/crio-164dacfab582ce4dd1f4cbef6c8f07665f5ddf4377cb445adb92fe627866a409 WatchSource:0}: Error finding container 164dacfab582ce4dd1f4cbef6c8f07665f5ddf4377cb445adb92fe627866a409: Status 404 returned error can't find the container with id 164dacfab582ce4dd1f4cbef6c8f07665f5ddf4377cb445adb92fe627866a409
Sep 10 23:08:41.872631 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:08:41.872601    2564 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 10 23:08:41.872631 ip-10-0-129-219 kubenswrapper[2564]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-xgj9h_kuadrant-system_37593210-7b63-459e-92cb-dd578993c392_0(eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-xgj9h to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28" Netns:"/var/run/netns/d18b3ab7-d146-4b10-a44b-32f0e57c8465" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-xgj9h;K8S_POD_INFRA_CONTAINER_ID=eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28;K8S_POD_UID=37593210-7b63-459e-92cb-dd578993c392" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h/37593210-7b63-459e-92cb-dd578993c392:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28 network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-7978cbdfbc-xgj9h" not found
Sep 10 23:08:41.872631 ip-10-0-129-219 kubenswrapper[2564]:         '
Sep 10 23:08:41.872631 ip-10-0-129-219 kubenswrapper[2564]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 10 23:08:41.872631 ip-10-0-129-219 kubenswrapper[2564]:  >
Sep 10 23:08:41.872843 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:08:41.872659    2564 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 10 23:08:41.872843 ip-10-0-129-219 kubenswrapper[2564]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-xgj9h_kuadrant-system_37593210-7b63-459e-92cb-dd578993c392_0(eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-xgj9h to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28" Netns:"/var/run/netns/d18b3ab7-d146-4b10-a44b-32f0e57c8465" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-xgj9h;K8S_POD_INFRA_CONTAINER_ID=eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28;K8S_POD_UID=37593210-7b63-459e-92cb-dd578993c392" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h/37593210-7b63-459e-92cb-dd578993c392:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h eb89015096890441a557274ed2b6d947f9d2c04b4ccef64610de44a95196be28 network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-7978cbdfbc-xgj9h" not found
Sep 10 23:08:41.872843 ip-10-0-129-219 kubenswrapper[2564]:         '
Sep 10 23:08:41.872843 ip-10-0-129-219 kubenswrapper[2564]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 10 23:08:41.872843 ip-10-0-129-219 kubenswrapper[2564]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:42.003082 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.003047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p" event={"ID":"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb","Type":"ContainerStarted","Data":"164dacfab582ce4dd1f4cbef6c8f07665f5ddf4377cb445adb92fe627866a409"}
Sep 10 23:08:42.003376 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.003093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:42.005307 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:08:42.005284    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" is forbidden: User \"system:node:ip-10-0-129-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-219.ec2.internal' and this object" podUID="37593210-7b63-459e-92cb-dd578993c392" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:42.010554 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.010535    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:42.013435 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:08:42.013411    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" is forbidden: User \"system:node:ip-10-0-129-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-219.ec2.internal' and this object" podUID="37593210-7b63-459e-92cb-dd578993c392" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:42.152920 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.152851    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37593210-7b63-459e-92cb-dd578993c392-extensions-socket-volume\") pod \"37593210-7b63-459e-92cb-dd578993c392\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") "
Sep 10 23:08:42.152920 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.152913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flnnn\" (UniqueName: \"kubernetes.io/projected/37593210-7b63-459e-92cb-dd578993c392-kube-api-access-flnnn\") pod \"37593210-7b63-459e-92cb-dd578993c392\" (UID: \"37593210-7b63-459e-92cb-dd578993c392\") "
Sep 10 23:08:42.153164 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.153130    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37593210-7b63-459e-92cb-dd578993c392-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37593210-7b63-459e-92cb-dd578993c392" (UID: "37593210-7b63-459e-92cb-dd578993c392"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:42.155279 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.155253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37593210-7b63-459e-92cb-dd578993c392-kube-api-access-flnnn" (OuterVolumeSpecName: "kube-api-access-flnnn") pod "37593210-7b63-459e-92cb-dd578993c392" (UID: "37593210-7b63-459e-92cb-dd578993c392"). InnerVolumeSpecName "kube-api-access-flnnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:42.199019 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.198992    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37593210-7b63-459e-92cb-dd578993c392" path="/var/lib/kubelet/pods/37593210-7b63-459e-92cb-dd578993c392/volumes"
Sep 10 23:08:42.253862 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.253834    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnnn\" (UniqueName: \"kubernetes.io/projected/37593210-7b63-459e-92cb-dd578993c392-kube-api-access-flnnn\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:08:42.253996 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:42.253864    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37593210-7b63-459e-92cb-dd578993c392-extensions-socket-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:08:43.008324 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:43.008286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:43.013418 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:08:43.013384    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-xgj9h\" is forbidden: User \"system:node:ip-10-0-129-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-219.ec2.internal' and this object" podUID="37593210-7b63-459e-92cb-dd578993c392" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-xgj9h"
Sep 10 23:08:46.028719 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:46.028674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p" event={"ID":"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb","Type":"ContainerStarted","Data":"e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6"}
Sep 10 23:08:46.029144 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:46.028847    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:08:46.052950 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:46.052903    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p" podStartSLOduration=2.605404163 podStartE2EDuration="6.052890669s" totalImagesPullingTime="3.447486506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:46.051610682 +0000 UTC m=+430.337631885" watchObservedRunningTime="2026-09-10 23:08:46.052890669 +0000 UTC m=+430.338911933"
Sep 10 23:08:57.036094 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:08:57.036064    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:09:09.099541 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.099461    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"]
Sep 10 23:09:09.100047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.099732    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p" podUID="b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" containerName="manager" containerID="cri-o://e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6" gracePeriod=10
Sep 10 23:09:09.338962 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.338936    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:09:09.443901 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.443874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9hkp\" (UniqueName: \"kubernetes.io/projected/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-kube-api-access-p9hkp\") pod \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") "
Sep 10 23:09:09.444023 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.443909    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-extensions-socket-volume\") pod \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\" (UID: \"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb\") "
Sep 10 23:09:09.444450 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.444423    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" (UID: "b0435be0-4cb5-4345-b9fc-1258d5c5e7cb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:09:09.446133 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.446105    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-kube-api-access-p9hkp" (OuterVolumeSpecName: "kube-api-access-p9hkp") pod "b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" (UID: "b0435be0-4cb5-4345-b9fc-1258d5c5e7cb"). InnerVolumeSpecName "kube-api-access-p9hkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:09:09.545180 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.545156    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9hkp\" (UniqueName: \"kubernetes.io/projected/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-kube-api-access-p9hkp\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:09:09.545180 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:09.545178    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb-extensions-socket-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:09:10.137182 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.137142    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" containerID="e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6" exitCode=0
Sep 10 23:09:10.137568 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.137189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p" event={"ID":"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb","Type":"ContainerDied","Data":"e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6"}
Sep 10 23:09:10.137568 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.137202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"
Sep 10 23:09:10.137568 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.137273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p" event={"ID":"b0435be0-4cb5-4345-b9fc-1258d5c5e7cb","Type":"ContainerDied","Data":"164dacfab582ce4dd1f4cbef6c8f07665f5ddf4377cb445adb92fe627866a409"}
Sep 10 23:09:10.137568 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.137305    2564 scope.go:117] "RemoveContainer" containerID="e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6"
Sep 10 23:09:10.145781 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.145764    2564 scope.go:117] "RemoveContainer" containerID="e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6"
Sep 10 23:09:10.146025 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:09:10.146007    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6\": container with ID starting with e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6 not found: ID does not exist" containerID="e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6"
Sep 10 23:09:10.146070 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.146032    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6"} err="failed to get container status \"e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6\": rpc error: code = NotFound desc = could not find container \"e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6\": container with ID starting with e8fe9b23f0e99412830ba450be52ac3048227c56a4e8f1a31e994aec196904d6 not found: ID does not exist"
Sep 10 23:09:10.165579 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.165553    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"]
Sep 10 23:09:10.169687 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.169666    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-tn68p"]
Sep 10 23:09:10.197060 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:10.197039    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" path="/var/lib/kubelet/pods/b0435be0-4cb5-4345-b9fc-1258d5c5e7cb/volumes"
Sep 10 23:09:27.109574 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.109538    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:27.110042 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.109883    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" containerName="manager"
Sep 10 23:09:27.110042 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.109896    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" containerName="manager"
Sep 10 23:09:27.110042 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.109948    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0435be0-4cb5-4345-b9fc-1258d5c5e7cb" containerName="manager"
Sep 10 23:09:27.112626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.112609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.115597 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.115574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6kc25\""
Sep 10 23:09:27.115894 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.115877    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 23:09:27.128423 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.128405    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:27.171974 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.171947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-config-file\") pod \"limitador-limitador-8588749977-998hb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") " pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.172112 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.171988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6c7m\" (UniqueName: \"kubernetes.io/projected/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-kube-api-access-p6c7m\") pod \"limitador-limitador-8588749977-998hb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") " pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.207002 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.206975    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:27.272645 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.272619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-config-file\") pod \"limitador-limitador-8588749977-998hb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") " pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.272798 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.272661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6c7m\" (UniqueName: \"kubernetes.io/projected/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-kube-api-access-p6c7m\") pod \"limitador-limitador-8588749977-998hb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") " pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.273288 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.273268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-config-file\") pod \"limitador-limitador-8588749977-998hb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") " pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.280907 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.280886    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6c7m\" (UniqueName: \"kubernetes.io/projected/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-kube-api-access-p6c7m\") pod \"limitador-limitador-8588749977-998hb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") " pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.421991 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.421968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:27.691563 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:27.691518    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:27.693842 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:09:27.693818    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa1bf8b_ce5b_4658_ad6e_bbafe0f433cb.slice/crio-652e57927d6490698650f01efd9df7293a4b3a9be31e969e091744b3440ad827 WatchSource:0}: Error finding container 652e57927d6490698650f01efd9df7293a4b3a9be31e969e091744b3440ad827: Status 404 returned error can't find the container with id 652e57927d6490698650f01efd9df7293a4b3a9be31e969e091744b3440ad827
Sep 10 23:09:28.219774 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:28.219739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-998hb" event={"ID":"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb","Type":"ContainerStarted","Data":"652e57927d6490698650f01efd9df7293a4b3a9be31e969e091744b3440ad827"}
Sep 10 23:09:29.231208 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:29.231171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-998hb" event={"ID":"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb","Type":"ContainerStarted","Data":"ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080"}
Sep 10 23:09:29.231603 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:29.231276    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:29.254515 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:29.254472    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-998hb" podStartSLOduration=1.107778179 podStartE2EDuration="2.254459602s" totalImagesPullingTime="1.146681423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:29.252931115 +0000 UTC m=+473.538952316" watchObservedRunningTime="2026-09-10 23:09:29.254459602 +0000 UTC m=+473.540480842"
Sep 10 23:09:40.236459 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.236424    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:40.262233 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.262206    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:40.278453 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.278421    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-998hb" podUID="caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" containerName="limitador" containerID="cri-o://ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080" gracePeriod=30
Sep 10 23:09:40.815526 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.815504    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:40.860141 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.860068    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6c7m\" (UniqueName: \"kubernetes.io/projected/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-kube-api-access-p6c7m\") pod \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") "
Sep 10 23:09:40.860141 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.860118    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-config-file\") pod \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\" (UID: \"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb\") "
Sep 10 23:09:40.860479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.860456    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-config-file" (OuterVolumeSpecName: "config-file") pod "caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" (UID: "caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:09:40.862120 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.862096    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-kube-api-access-p6c7m" (OuterVolumeSpecName: "kube-api-access-p6c7m") pod "caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" (UID: "caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb"). InnerVolumeSpecName "kube-api-access-p6c7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:09:40.961072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.961047    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6c7m\" (UniqueName: \"kubernetes.io/projected/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-kube-api-access-p6c7m\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:09:40.961072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:40.961070    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb-config-file\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:09:41.283730 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.283675    2564 generic.go:356] "Generic (PLEG): container finished" podID="caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" containerID="ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080" exitCode=0
Sep 10 23:09:41.284196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.283741    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-998hb"
Sep 10 23:09:41.284196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.283763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-998hb" event={"ID":"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb","Type":"ContainerDied","Data":"ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080"}
Sep 10 23:09:41.284196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.283819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-998hb" event={"ID":"caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb","Type":"ContainerDied","Data":"652e57927d6490698650f01efd9df7293a4b3a9be31e969e091744b3440ad827"}
Sep 10 23:09:41.284196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.283844    2564 scope.go:117] "RemoveContainer" containerID="ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080"
Sep 10 23:09:41.292322 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.292307    2564 scope.go:117] "RemoveContainer" containerID="ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080"
Sep 10 23:09:41.292613 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:09:41.292592    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080\": container with ID starting with ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080 not found: ID does not exist" containerID="ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080"
Sep 10 23:09:41.292682 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.292617    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080"} err="failed to get container status \"ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080\": rpc error: code = NotFound desc = could not find container \"ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080\": container with ID starting with ab8c02a14b8974d71d21b1cdc187aed2336715480d5a9e17899e91385eaef080 not found: ID does not exist"
Sep 10 23:09:41.316829 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.316805    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:41.326368 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:41.326343    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-998hb"]
Sep 10 23:09:42.203188 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.203145    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" path="/var/lib/kubelet/pods/caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb/volumes"
Sep 10 23:09:42.884028 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.883992    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"]
Sep 10 23:09:42.884438 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.884314    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" containerName="limitador"
Sep 10 23:09:42.884438 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.884327    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" containerName="limitador"
Sep 10 23:09:42.884438 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.884384    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa1bf8b-ce5b-4658-ad6e-bbafe0f433cb" containerName="limitador"
Sep 10 23:09:42.888573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.888550    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:42.891190 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.891153    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:09:42.891326 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.891185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bzpj7\""
Sep 10 23:09:42.892165 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.892134    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 23:09:42.892165 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.892134    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:09:42.898231 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.898206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"]
Sep 10 23:09:42.971736 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.971688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxd5\" (UniqueName: \"kubernetes.io/projected/a84e8cf6-23f7-4e7c-87d6-26dce1e64a40-kube-api-access-tsxd5\") pod \"postgres-69fb4594cf-rd8nk\" (UID: \"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:42.971838 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:42.971775    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a84e8cf6-23f7-4e7c-87d6-26dce1e64a40-data\") pod \"postgres-69fb4594cf-rd8nk\" (UID: \"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:43.072172 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:43.072139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxd5\" (UniqueName: \"kubernetes.io/projected/a84e8cf6-23f7-4e7c-87d6-26dce1e64a40-kube-api-access-tsxd5\") pod \"postgres-69fb4594cf-rd8nk\" (UID: \"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:43.072272 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:43.072177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a84e8cf6-23f7-4e7c-87d6-26dce1e64a40-data\") pod \"postgres-69fb4594cf-rd8nk\" (UID: \"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:43.072583 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:43.072568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a84e8cf6-23f7-4e7c-87d6-26dce1e64a40-data\") pod \"postgres-69fb4594cf-rd8nk\" (UID: \"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:43.081224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:43.081205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxd5\" (UniqueName: \"kubernetes.io/projected/a84e8cf6-23f7-4e7c-87d6-26dce1e64a40-kube-api-access-tsxd5\") pod \"postgres-69fb4594cf-rd8nk\" (UID: \"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:43.199813 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:43.199779    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:43.467948 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:43.467925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"]
Sep 10 23:09:43.470032 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:09:43.470004    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84e8cf6_23f7_4e7c_87d6_26dce1e64a40.slice/crio-f936e82be8515edc1e38fcc648ed3fb5d292181b1df9c59c52979b09364a201a WatchSource:0}: Error finding container f936e82be8515edc1e38fcc648ed3fb5d292181b1df9c59c52979b09364a201a: Status 404 returned error can't find the container with id f936e82be8515edc1e38fcc648ed3fb5d292181b1df9c59c52979b09364a201a
Sep 10 23:09:44.299567 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:44.299536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk" event={"ID":"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40","Type":"ContainerStarted","Data":"f936e82be8515edc1e38fcc648ed3fb5d292181b1df9c59c52979b09364a201a"}
Sep 10 23:09:49.323144 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:49.323111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk" event={"ID":"a84e8cf6-23f7-4e7c-87d6-26dce1e64a40","Type":"ContainerStarted","Data":"4cc97a648de1ca191c6512910ab859c5d33d0d35b1ff5f4649cf08a68411c7a8"}
Sep 10 23:09:49.323506 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:49.323248    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:09:49.341936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:49.341869    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk" podStartSLOduration=2.443814711 podStartE2EDuration="7.341856102s" totalImagesPullingTime="4.898041391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:49.340748006 +0000 UTC m=+493.626769199" watchObservedRunningTime="2026-09-10 23:09:49.341856102 +0000 UTC m=+493.627877303"
Sep 10 23:09:55.355796 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:09:55.355769    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rd8nk"
Sep 10 23:10:04.127781 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.127750    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66dc98c474-svfpm"]
Sep 10 23:10:04.140280 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.140259    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.142727 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.142680    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:10:04.143047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.143018    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:10:04.143484 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.143462    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fg9zd\""
Sep 10 23:10:04.143600 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.143519    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:10:04.144196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.144170    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66dc98c474-svfpm"]
Sep 10 23:10:04.202405 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.202383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-webhook-cert\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.202532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.202428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.202532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.202446    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6f82\" (UniqueName: \"kubernetes.io/projected/d5a163f6-e7eb-4192-ac15-cf3ea0573804-kube-api-access-r6f82\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.303438 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.303414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.303539 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.303445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6f82\" (UniqueName: \"kubernetes.io/projected/d5a163f6-e7eb-4192-ac15-cf3ea0573804-kube-api-access-r6f82\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.303539 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.303498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-webhook-cert\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.303616 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:10:04.303545    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 23:10:04.303616 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:10:04.303601    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs podName:d5a163f6-e7eb-4192-ac15-cf3ea0573804 nodeName:}" failed. No retries permitted until 2026-09-10 23:10:04.803585365 +0000 UTC m=+509.089606545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs") pod "maas-controller-66dc98c474-svfpm" (UID: "d5a163f6-e7eb-4192-ac15-cf3ea0573804") : secret "maas-controller-metrics-tls" not found
Sep 10 23:10:04.305901 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.305881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-webhook-cert\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.309895 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.309876    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-gq2vz"]
Sep 10 23:10:04.312049 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.312029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.316483 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.316462    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6f82\" (UniqueName: \"kubernetes.io/projected/d5a163f6-e7eb-4192-ac15-cf3ea0573804-kube-api-access-r6f82\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.323818 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.323796    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-gq2vz"]
Sep 10 23:10:04.404209 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.404131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6nw\" (UniqueName: \"kubernetes.io/projected/0e2ec996-a933-49ba-8620-fb063c377f28-kube-api-access-6d6nw\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.404209 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.404203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-webhook-cert\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.404378 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.404234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-metrics-certs\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.504967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.504934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6nw\" (UniqueName: \"kubernetes.io/projected/0e2ec996-a933-49ba-8620-fb063c377f28-kube-api-access-6d6nw\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.505133 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.504986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-webhook-cert\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.505133 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.505008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-metrics-certs\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.507362 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.507341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-webhook-cert\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.507439 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.507393    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-metrics-certs\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.514309 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.514287    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6nw\" (UniqueName: \"kubernetes.io/projected/0e2ec996-a933-49ba-8620-fb063c377f28-kube-api-access-6d6nw\") pod \"maas-controller-7ff55dfbf9-gq2vz\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") " pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.629847 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.629818    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:04.807125 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.807085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.809389 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.809360    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs\") pod \"maas-controller-66dc98c474-svfpm\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") " pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:04.899057 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:04.899034    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-gq2vz"]
Sep 10 23:10:04.900969 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:10:04.900940    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2ec996_a933_49ba_8620_fb063c377f28.slice/crio-9c7121bb9eba31898c9e8e4847d09b503707b12dedc18001c37d364d2fb33472 WatchSource:0}: Error finding container 9c7121bb9eba31898c9e8e4847d09b503707b12dedc18001c37d364d2fb33472: Status 404 returned error can't find the container with id 9c7121bb9eba31898c9e8e4847d09b503707b12dedc18001c37d364d2fb33472
Sep 10 23:10:05.051918 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:05.051846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:05.321020 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:05.320857    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66dc98c474-svfpm"]
Sep 10 23:10:05.323417 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:10:05.323393    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a163f6_e7eb_4192_ac15_cf3ea0573804.slice/crio-85b43ac5de27087a2f1897fea102025219a911696cf8751630bea84602e14e01 WatchSource:0}: Error finding container 85b43ac5de27087a2f1897fea102025219a911696cf8751630bea84602e14e01: Status 404 returned error can't find the container with id 85b43ac5de27087a2f1897fea102025219a911696cf8751630bea84602e14e01
Sep 10 23:10:05.392852 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:05.392822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66dc98c474-svfpm" event={"ID":"d5a163f6-e7eb-4192-ac15-cf3ea0573804","Type":"ContainerStarted","Data":"85b43ac5de27087a2f1897fea102025219a911696cf8751630bea84602e14e01"}
Sep 10 23:10:05.394003 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:05.393977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz" event={"ID":"0e2ec996-a933-49ba-8620-fb063c377f28","Type":"ContainerStarted","Data":"9c7121bb9eba31898c9e8e4847d09b503707b12dedc18001c37d364d2fb33472"}
Sep 10 23:10:08.413885 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:08.413840    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66dc98c474-svfpm" event={"ID":"d5a163f6-e7eb-4192-ac15-cf3ea0573804","Type":"ContainerStarted","Data":"c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41"}
Sep 10 23:10:08.414334 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:08.413978    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:08.415360 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:08.415332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz" event={"ID":"0e2ec996-a933-49ba-8620-fb063c377f28","Type":"ContainerStarted","Data":"9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77"}
Sep 10 23:10:08.415487 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:08.415379    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:08.449472 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:08.449426    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66dc98c474-svfpm" podStartSLOduration=2.231279687 podStartE2EDuration="4.449413436s" totalImagesPullingTime="2.218133749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:10:08.447379565 +0000 UTC m=+512.733400765" watchObservedRunningTime="2026-09-10 23:10:08.449413436 +0000 UTC m=+512.735434636"
Sep 10 23:10:08.478373 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:08.478325    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz" podStartSLOduration=1.846720414 podStartE2EDuration="4.478309507s" totalImagesPullingTime="2.631589093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:10:08.477854219 +0000 UTC m=+512.763875433" watchObservedRunningTime="2026-09-10 23:10:08.478309507 +0000 UTC m=+512.764330708"
Sep 10 23:10:19.423240 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.423207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:19.423595 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.423283    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:10:19.497514 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.497482    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66dc98c474-svfpm"]
Sep 10 23:10:19.497727 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.497683    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66dc98c474-svfpm" podUID="d5a163f6-e7eb-4192-ac15-cf3ea0573804" containerName="manager" containerID="cri-o://c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41" gracePeriod=30
Sep 10 23:10:19.843156 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.843131    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:19.920058 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.920025    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs\") pod \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") "
Sep 10 23:10:19.920183 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.920092    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6f82\" (UniqueName: \"kubernetes.io/projected/d5a163f6-e7eb-4192-ac15-cf3ea0573804-kube-api-access-r6f82\") pod \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") "
Sep 10 23:10:19.920233 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.920217    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-webhook-cert\") pod \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\" (UID: \"d5a163f6-e7eb-4192-ac15-cf3ea0573804\") "
Sep 10 23:10:19.922179 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.922154    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5a163f6-e7eb-4192-ac15-cf3ea0573804" (UID: "d5a163f6-e7eb-4192-ac15-cf3ea0573804"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:10:19.922302 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.922213    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a163f6-e7eb-4192-ac15-cf3ea0573804-kube-api-access-r6f82" (OuterVolumeSpecName: "kube-api-access-r6f82") pod "d5a163f6-e7eb-4192-ac15-cf3ea0573804" (UID: "d5a163f6-e7eb-4192-ac15-cf3ea0573804"). InnerVolumeSpecName "kube-api-access-r6f82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:10:19.922302 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:19.922247    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d5a163f6-e7eb-4192-ac15-cf3ea0573804" (UID: "d5a163f6-e7eb-4192-ac15-cf3ea0573804"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:10:20.021521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.021458    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:10:20.021521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.021484    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6f82\" (UniqueName: \"kubernetes.io/projected/d5a163f6-e7eb-4192-ac15-cf3ea0573804-kube-api-access-r6f82\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:10:20.021521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.021496    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5a163f6-e7eb-4192-ac15-cf3ea0573804-webhook-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:10:20.472759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.472727    2564 generic.go:356] "Generic (PLEG): container finished" podID="d5a163f6-e7eb-4192-ac15-cf3ea0573804" containerID="c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41" exitCode=0
Sep 10 23:10:20.473199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.472786    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66dc98c474-svfpm"
Sep 10 23:10:20.473199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.472799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66dc98c474-svfpm" event={"ID":"d5a163f6-e7eb-4192-ac15-cf3ea0573804","Type":"ContainerDied","Data":"c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41"}
Sep 10 23:10:20.473199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.472848    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66dc98c474-svfpm" event={"ID":"d5a163f6-e7eb-4192-ac15-cf3ea0573804","Type":"ContainerDied","Data":"85b43ac5de27087a2f1897fea102025219a911696cf8751630bea84602e14e01"}
Sep 10 23:10:20.473199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.472869    2564 scope.go:117] "RemoveContainer" containerID="c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41"
Sep 10 23:10:20.480570 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.480554    2564 scope.go:117] "RemoveContainer" containerID="c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41"
Sep 10 23:10:20.480847 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:10:20.480822    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41\": container with ID starting with c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41 not found: ID does not exist" containerID="c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41"
Sep 10 23:10:20.480908 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.480855    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41"} err="failed to get container status \"c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41\": rpc error: code = NotFound desc = could not find container \"c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41\": container with ID starting with c112360f765099907fda8c0ca60ae95deac0f241042299fb2413037c6e91aa41 not found: ID does not exist"
Sep 10 23:10:20.492176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.492150    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66dc98c474-svfpm"]
Sep 10 23:10:20.496485 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:20.496465    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66dc98c474-svfpm"]
Sep 10 23:10:22.198290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:22.198250    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a163f6-e7eb-4192-ac15-cf3ea0573804" path="/var/lib/kubelet/pods/d5a163f6-e7eb-4192-ac15-cf3ea0573804/volumes"
Sep 10 23:10:41.163081 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.163040    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-htw9m"]
Sep 10 23:10:41.163463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.163370    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a163f6-e7eb-4192-ac15-cf3ea0573804" containerName="manager"
Sep 10 23:10:41.163463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.163382    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a163f6-e7eb-4192-ac15-cf3ea0573804" containerName="manager"
Sep 10 23:10:41.163533 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.163477    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a163f6-e7eb-4192-ac15-cf3ea0573804" containerName="manager"
Sep 10 23:10:41.167925 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.167905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.171503 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.171483    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 23:10:41.171931 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.171912    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:10:41.177223 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.177202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5kfc5\""
Sep 10 23:10:41.186546 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.186524    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-htw9m"]
Sep 10 23:10:41.276630 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.276601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-metrics-certs\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.276794 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.276740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-maas-api-tls\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.276794 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.276760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/d6339823-f27f-4e55-9c19-26ab7a5bb68b-kube-api-access-ggxcl\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.276928 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.276802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6339823-f27f-4e55-9c19-26ab7a5bb68b-openshift-service-ca\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.378182 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.378153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6339823-f27f-4e55-9c19-26ab7a5bb68b-openshift-service-ca\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.378295 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.378207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-metrics-certs\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.378295 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.378233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-maas-api-tls\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.378295 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.378249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/d6339823-f27f-4e55-9c19-26ab7a5bb68b-kube-api-access-ggxcl\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.378829 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.378808    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6339823-f27f-4e55-9c19-26ab7a5bb68b-openshift-service-ca\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.380784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.380762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-maas-api-tls\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.380873 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.380818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-metrics-certs\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.388031 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.388007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/d6339823-f27f-4e55-9c19-26ab7a5bb68b-kube-api-access-ggxcl\") pod \"maas-api-777777df97-htw9m\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") " pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.478189 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.478160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:41.755357 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.755282    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-htw9m"]
Sep 10 23:10:41.758137 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:10:41.758109    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6339823_f27f_4e55_9c19_26ab7a5bb68b.slice/crio-c303dd0902faa9193f70e6a23b82c0080dd19dbbb65f17dffa8f3093a2842554 WatchSource:0}: Error finding container c303dd0902faa9193f70e6a23b82c0080dd19dbbb65f17dffa8f3093a2842554: Status 404 returned error can't find the container with id c303dd0902faa9193f70e6a23b82c0080dd19dbbb65f17dffa8f3093a2842554
Sep 10 23:10:41.826756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.826733    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"]
Sep 10 23:10:41.831120 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.831098    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:41.834424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.834403    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hzhzk\""
Sep 10 23:10:41.834812 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.834786    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 23:10:41.846390 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.846369    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"]
Sep 10 23:10:41.882905 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.882880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6f4k\" (UniqueName: \"kubernetes.io/projected/0244d17f-d128-47dd-a306-7593b8f2191e-kube-api-access-c6f4k\") pod \"payload-pre-processing-59fbc47bf8-bdnnx\" (UID: \"0244d17f-d128-47dd-a306-7593b8f2191e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:41.883003 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.882925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0244d17f-d128-47dd-a306-7593b8f2191e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bdnnx\" (UID: \"0244d17f-d128-47dd-a306-7593b8f2191e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:41.984192 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.984153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6f4k\" (UniqueName: \"kubernetes.io/projected/0244d17f-d128-47dd-a306-7593b8f2191e-kube-api-access-c6f4k\") pod \"payload-pre-processing-59fbc47bf8-bdnnx\" (UID: \"0244d17f-d128-47dd-a306-7593b8f2191e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:41.984333 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.984224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0244d17f-d128-47dd-a306-7593b8f2191e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bdnnx\" (UID: \"0244d17f-d128-47dd-a306-7593b8f2191e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:41.984903 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.984884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0244d17f-d128-47dd-a306-7593b8f2191e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bdnnx\" (UID: \"0244d17f-d128-47dd-a306-7593b8f2191e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:41.993355 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:41.993328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6f4k\" (UniqueName: \"kubernetes.io/projected/0244d17f-d128-47dd-a306-7593b8f2191e-kube-api-access-c6f4k\") pod \"payload-pre-processing-59fbc47bf8-bdnnx\" (UID: \"0244d17f-d128-47dd-a306-7593b8f2191e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:42.142233 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:42.142167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:42.416913 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:42.416854    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"]
Sep 10 23:10:42.420493 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:10:42.420464    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0244d17f_d128_47dd_a306_7593b8f2191e.slice/crio-f173f4a8165507d12ec81fa627d435fc7eb237cc81d62f5791cc82586af16c32 WatchSource:0}: Error finding container f173f4a8165507d12ec81fa627d435fc7eb237cc81d62f5791cc82586af16c32: Status 404 returned error can't find the container with id f173f4a8165507d12ec81fa627d435fc7eb237cc81d62f5791cc82586af16c32
Sep 10 23:10:42.576097 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:42.576053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m" event={"ID":"d6339823-f27f-4e55-9c19-26ab7a5bb68b","Type":"ContainerStarted","Data":"c303dd0902faa9193f70e6a23b82c0080dd19dbbb65f17dffa8f3093a2842554"}
Sep 10 23:10:42.577616 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:42.577580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx" event={"ID":"0244d17f-d128-47dd-a306-7593b8f2191e","Type":"ContainerStarted","Data":"f173f4a8165507d12ec81fa627d435fc7eb237cc81d62f5791cc82586af16c32"}
Sep 10 23:10:44.587749 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:44.587688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m" event={"ID":"d6339823-f27f-4e55-9c19-26ab7a5bb68b","Type":"ContainerStarted","Data":"a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710"}
Sep 10 23:10:44.588164 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:44.587776    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:44.589266 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:44.589242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx" event={"ID":"0244d17f-d128-47dd-a306-7593b8f2191e","Type":"ContainerStarted","Data":"79d3378bdba482671c91355887e8d3d0b4be50278a4c23b969308d3a6dcacaa7"}
Sep 10 23:10:44.589379 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:44.589285    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:10:44.611633 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:44.611588    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m" podStartSLOduration=1.8214422620000001 podStartE2EDuration="3.61156931s" totalImagesPullingTime="1.790127048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:10:44.608965252 +0000 UTC m=+548.894986452" watchObservedRunningTime="2026-09-10 23:10:44.61156931 +0000 UTC m=+548.897590511"
Sep 10 23:10:44.628541 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:44.628501    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx" podStartSLOduration=2.01261431 podStartE2EDuration="3.628488215s" totalImagesPullingTime="1.615873905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:10:44.62712483 +0000 UTC m=+548.913146040" watchObservedRunningTime="2026-09-10 23:10:44.628488215 +0000 UTC m=+548.914509416"
Sep 10 23:10:50.600335 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:50.600308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:10:55.597535 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:10:55.597503    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bdnnx"
Sep 10 23:11:22.361149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.361117    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"]
Sep 10 23:11:22.366648 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.366628    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.369416 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.369378    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:11:22.369526 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.369499    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnpmw\""
Sep 10 23:11:22.370442 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.370423    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 23:11:22.370540 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.370453    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:11:22.375564 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.375541    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"]
Sep 10 23:11:22.395360 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.395325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.395475 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.395386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e1d1a4-c40f-4254-bc70-8463956df8bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.395475 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.395415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.395475 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.395435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.395637 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.395481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.395637 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.395523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kps95\" (UniqueName: \"kubernetes.io/projected/38e1d1a4-c40f-4254-bc70-8463956df8bd-kube-api-access-kps95\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496300 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e1d1a4-c40f-4254-bc70-8463956df8bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496300 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496481 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496481 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496481 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kps95\" (UniqueName: \"kubernetes.io/projected/38e1d1a4-c40f-4254-bc70-8463956df8bd-kube-api-access-kps95\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496481 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496770 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496815 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.496849 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.496818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.498748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.498698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38e1d1a4-c40f-4254-bc70-8463956df8bd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.498829 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.498766    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38e1d1a4-c40f-4254-bc70-8463956df8bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.506103 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.506084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kps95\" (UniqueName: \"kubernetes.io/projected/38e1d1a4-c40f-4254-bc70-8463956df8bd-kube-api-access-kps95\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2pcjv\" (UID: \"38e1d1a4-c40f-4254-bc70-8463956df8bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.677975 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.677953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:22.951751 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:22.951728    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"]
Sep 10 23:11:22.953956 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:11:22.953918    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e1d1a4_c40f_4254_bc70_8463956df8bd.slice/crio-36840c24d7f0da2998fa4a26196d68e46c9348203248db45a7ec76e21613b01e WatchSource:0}: Error finding container 36840c24d7f0da2998fa4a26196d68e46c9348203248db45a7ec76e21613b01e: Status 404 returned error can't find the container with id 36840c24d7f0da2998fa4a26196d68e46c9348203248db45a7ec76e21613b01e
Sep 10 23:11:23.773329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:23.773293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv" event={"ID":"38e1d1a4-c40f-4254-bc70-8463956df8bd","Type":"ContainerStarted","Data":"36840c24d7f0da2998fa4a26196d68e46c9348203248db45a7ec76e21613b01e"}
Sep 10 23:11:24.779049 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:24.779014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv" event={"ID":"38e1d1a4-c40f-4254-bc70-8463956df8bd","Type":"ContainerStarted","Data":"289a185ff93f302a4bf4055c4b64f5804ca047881dd588dc5564aaa1ba298e4e"}
Sep 10 23:11:24.797894 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:24.797841    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv" podStartSLOduration=1.820471812 podStartE2EDuration="2.797819886s" totalImagesPullingTime="977.348074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:11:24.797786814 +0000 UTC m=+589.083839896" watchObservedRunningTime="2026-09-10 23:11:24.797819886 +0000 UTC m=+589.083841089"
Sep 10 23:11:32.678865 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:32.678830    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:32.679266 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:32.678890    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:32.691258 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:32.691228    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:32.826208 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:32.826179    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2pcjv"
Sep 10 23:11:34.253392 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.253358    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"]
Sep 10 23:11:34.256890 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.256868    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.259456 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.259435    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 23:11:34.267336 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.267313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"]
Sep 10 23:11:34.285040 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.285013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.285149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.285044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aaaca42-9bdf-4d23-85e2-444b921ea26c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.285149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.285122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zm4m\" (UniqueName: \"kubernetes.io/projected/5aaaca42-9bdf-4d23-85e2-444b921ea26c-kube-api-access-4zm4m\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.285268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.285180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.285268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.285235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.285268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.285263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386296 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aaaca42-9bdf-4d23-85e2-444b921ea26c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zm4m\" (UniqueName: \"kubernetes.io/projected/5aaaca42-9bdf-4d23-85e2-444b921ea26c-kube-api-access-4zm4m\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386665 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386778 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.386826 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.386756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.388741 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.388720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aaaca42-9bdf-4d23-85e2-444b921ea26c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.388867 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.388849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aaaca42-9bdf-4d23-85e2-444b921ea26c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.395163 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.395139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zm4m\" (UniqueName: \"kubernetes.io/projected/5aaaca42-9bdf-4d23-85e2-444b921ea26c-kube-api-access-4zm4m\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj\" (UID: \"5aaaca42-9bdf-4d23-85e2-444b921ea26c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.567660 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.567597    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:34.840590 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:34.840566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"]
Sep 10 23:11:34.843212 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:11:34.843185    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aaaca42_9bdf_4d23_85e2_444b921ea26c.slice/crio-f55d1aef4a15b6483a4a59e0b1feb5180579fba6b2da210eeaed2dba23c1c425 WatchSource:0}: Error finding container f55d1aef4a15b6483a4a59e0b1feb5180579fba6b2da210eeaed2dba23c1c425: Status 404 returned error can't find the container with id f55d1aef4a15b6483a4a59e0b1feb5180579fba6b2da210eeaed2dba23c1c425
Sep 10 23:11:35.836459 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:35.836418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj" event={"ID":"5aaaca42-9bdf-4d23-85e2-444b921ea26c","Type":"ContainerStarted","Data":"7c1c76d033ff2d839634c2a446cf94e38a6abba275ea43ce3a36d9ff29b0e643"}
Sep 10 23:11:35.836934 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:35.836497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj" event={"ID":"5aaaca42-9bdf-4d23-85e2-444b921ea26c","Type":"ContainerStarted","Data":"f55d1aef4a15b6483a4a59e0b1feb5180579fba6b2da210eeaed2dba23c1c425"}
Sep 10 23:11:35.856437 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:35.856385    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj" podStartSLOduration=1.654262658 podStartE2EDuration="1.856371115s" totalImagesPullingTime="202.108457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:11:35.855147629 +0000 UTC m=+600.141168834" watchObservedRunningTime="2026-09-10 23:11:35.856371115 +0000 UTC m=+600.142392349"
Sep 10 23:11:36.121323 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:36.121249    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:11:36.122757 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:36.122734    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:11:44.568732 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:44.568689    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:44.571309 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:44.568759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:44.581014 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:44.580984    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:44.887329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:44.887249    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj"
Sep 10 23:11:58.156613 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.156579    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"]
Sep 10 23:11:58.167795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.167765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.170378 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.170342    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 23:11:58.174150 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.174127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"]
Sep 10 23:11:58.274893 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.274854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fd44f-d0c1-4246-b6a5-011169537b37-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.275076 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.275010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.275076 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.275054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.275186 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.275154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.275261 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.275233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.275312 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.275283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtff\" (UniqueName: \"kubernetes.io/projected/626fd44f-d0c1-4246-b6a5-011169537b37-kube-api-access-5vtff\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.376617 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.376617 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.376826 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.376826 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtff\" (UniqueName: \"kubernetes.io/projected/626fd44f-d0c1-4246-b6a5-011169537b37-kube-api-access-5vtff\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.376826 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fd44f-d0c1-4246-b6a5-011169537b37-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.376929 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.377012 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.376990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.377058 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.377027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.377241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.377224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.378886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.378868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fd44f-d0c1-4246-b6a5-011169537b37-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.379191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.379174    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fd44f-d0c1-4246-b6a5-011169537b37-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.384810 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.384790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtff\" (UniqueName: \"kubernetes.io/projected/626fd44f-d0c1-4246-b6a5-011169537b37-kube-api-access-5vtff\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zssvn\" (UID: \"626fd44f-d0c1-4246-b6a5-011169537b37\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.482086 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.482033    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:11:58.766749 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.766720    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"]
Sep 10 23:11:58.768554 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:11:58.768532    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626fd44f_d0c1_4246_b6a5_011169537b37.slice/crio-b34f93d811b246d04fea52d67f5b7f25542b8774839b58349da7cead094df42c WatchSource:0}: Error finding container b34f93d811b246d04fea52d67f5b7f25542b8774839b58349da7cead094df42c: Status 404 returned error can't find the container with id b34f93d811b246d04fea52d67f5b7f25542b8774839b58349da7cead094df42c
Sep 10 23:11:58.769961 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.769944    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:11:58.942672 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:58.942646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn" event={"ID":"626fd44f-d0c1-4246-b6a5-011169537b37","Type":"ContainerStarted","Data":"b34f93d811b246d04fea52d67f5b7f25542b8774839b58349da7cead094df42c"}
Sep 10 23:11:59.949422 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:59.949387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn" event={"ID":"626fd44f-d0c1-4246-b6a5-011169537b37","Type":"ContainerStarted","Data":"197ef45494a6383f6c2aecc1397ea767bc3c83957907a306b4e814effeefd917"}
Sep 10 23:11:59.972396 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:11:59.972335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn" podStartSLOduration=1.8020757330000001 podStartE2EDuration="1.972320226s" totalImagesPullingTime="170.244493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:11:59.972281739 +0000 UTC m=+624.258302944" watchObservedRunningTime="2026-09-10 23:11:59.972320226 +0000 UTC m=+624.258341424"
Sep 10 23:12:08.483012 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:08.482980    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:12:08.483380 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:08.483042    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:12:08.495391 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:08.495360    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:12:09.003351 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:09.003324    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zssvn"
Sep 10 23:12:10.555925 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.555885    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"]
Sep 10 23:12:10.562461 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.562440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.564890 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.564871    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 23:12:10.570521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.570368    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"]
Sep 10 23:12:10.666856 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.666829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.667007 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.666867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhmf\" (UniqueName: \"kubernetes.io/projected/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-kube-api-access-qkhmf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.667007 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.666941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.667007 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.666979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.667160 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.667012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.667160 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.667046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.767886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.767856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768021 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.767942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768021 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.767979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhmf\" (UniqueName: \"kubernetes.io/projected/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-kube-api-access-qkhmf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768129 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.768024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768129 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.768051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768129 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.768079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768423 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.768407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768423 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.768413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.768532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.768465    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.770220 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.770202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.770423 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.770405    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.775947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.775924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhmf\" (UniqueName: \"kubernetes.io/projected/e9c08499-f7a6-4d03-9092-7a6d1c79ee94-kube-api-access-qkhmf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd\" (UID: \"e9c08499-f7a6-4d03-9092-7a6d1c79ee94\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:10.874845 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:10.874793    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:11.142329 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:11.142304    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"]
Sep 10 23:12:11.144319 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:12:11.144293    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c08499_f7a6_4d03_9092_7a6d1c79ee94.slice/crio-25539741e200bcfc90e1363fe81f2844498100e7bb177416366e91c59fa7cef6 WatchSource:0}: Error finding container 25539741e200bcfc90e1363fe81f2844498100e7bb177416366e91c59fa7cef6: Status 404 returned error can't find the container with id 25539741e200bcfc90e1363fe81f2844498100e7bb177416366e91c59fa7cef6
Sep 10 23:12:12.007603 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:12.007568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd" event={"ID":"e9c08499-f7a6-4d03-9092-7a6d1c79ee94","Type":"ContainerStarted","Data":"d17e4d090b33a53ea77f5ba3dac3de9041025a9c768a590b2dfa451e81b602e7"}
Sep 10 23:12:12.008087 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:12.007626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd" event={"ID":"e9c08499-f7a6-4d03-9092-7a6d1c79ee94","Type":"ContainerStarted","Data":"25539741e200bcfc90e1363fe81f2844498100e7bb177416366e91c59fa7cef6"}
Sep 10 23:12:12.025132 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:12.025076    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd" podStartSLOduration=1.8161540619999998 podStartE2EDuration="2.025061263s" totalImagesPullingTime="208.907201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:12.024201022 +0000 UTC m=+636.310222218" watchObservedRunningTime="2026-09-10 23:12:12.025061263 +0000 UTC m=+636.311082464"
Sep 10 23:12:20.875515 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:20.875455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:20.875949 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:20.875549    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:20.887462 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:20.887432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:21.059556 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:21.059531    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd"
Sep 10 23:12:22.645388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.645359    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"]
Sep 10 23:12:22.649113 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.649088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.651289 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.651266    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 23:12:22.660047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.660023    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"]
Sep 10 23:12:22.761572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.761543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a18e2e9-8858-45b0-9f54-264b4912eaa0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.761572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.761577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.761777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.761594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rd6j\" (UniqueName: \"kubernetes.io/projected/8a18e2e9-8858-45b0-9f54-264b4912eaa0-kube-api-access-4rd6j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.761777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.761659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.761777 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.761750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.761874 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.761792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.862686 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.862659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a18e2e9-8858-45b0-9f54-264b4912eaa0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.862870 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.862726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.862870 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.862754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rd6j\" (UniqueName: \"kubernetes.io/projected/8a18e2e9-8858-45b0-9f54-264b4912eaa0-kube-api-access-4rd6j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.862870 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.862783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.862870 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.862814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.862870 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.862866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.863293 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.863260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.863293 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.863279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.863457 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.863310    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.864931 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.864912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a18e2e9-8858-45b0-9f54-264b4912eaa0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.865184 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.865165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a18e2e9-8858-45b0-9f54-264b4912eaa0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.872015 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.871995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rd6j\" (UniqueName: \"kubernetes.io/projected/8a18e2e9-8858-45b0-9f54-264b4912eaa0-kube-api-access-4rd6j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c6rgj\" (UID: \"8a18e2e9-8858-45b0-9f54-264b4912eaa0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:22.962951 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:22.962927    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:23.235570 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:23.235548    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"]
Sep 10 23:12:23.237100 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:12:23.237073    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a18e2e9_8858_45b0_9f54_264b4912eaa0.slice/crio-10caffe55dd44197582cbe839473b6fc2988214a80f64cd65a9121b15aed0b69 WatchSource:0}: Error finding container 10caffe55dd44197582cbe839473b6fc2988214a80f64cd65a9121b15aed0b69: Status 404 returned error can't find the container with id 10caffe55dd44197582cbe839473b6fc2988214a80f64cd65a9121b15aed0b69
Sep 10 23:12:24.062784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:24.062744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj" event={"ID":"8a18e2e9-8858-45b0-9f54-264b4912eaa0","Type":"ContainerStarted","Data":"5a8d9fe9a40e1320cc9d44714fc2e9e92601f4a7f29e5e5eebba27eec42ac471"}
Sep 10 23:12:24.063136 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:24.062827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj" event={"ID":"8a18e2e9-8858-45b0-9f54-264b4912eaa0","Type":"ContainerStarted","Data":"10caffe55dd44197582cbe839473b6fc2988214a80f64cd65a9121b15aed0b69"}
Sep 10 23:12:24.080552 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:24.080504    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj" podStartSLOduration=1.897755165 podStartE2EDuration="2.080488864s" totalImagesPullingTime="182.733699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:24.079626895 +0000 UTC m=+648.365648097" watchObservedRunningTime="2026-09-10 23:12:24.080488864 +0000 UTC m=+648.366510095"
Sep 10 23:12:32.964026 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:32.963996    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:32.964475 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:32.964050    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:32.976805 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:32.976781    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:12:33.115678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:12:33.115654    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c6rgj"
Sep 10 23:14:11.847759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.847669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"]
Sep 10 23:14:11.854999 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.854970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:11.857772 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.857749    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7025b9-serving-cert\""
Sep 10 23:14:11.863564 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.863535    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"]
Sep 10 23:14:11.929392 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.929366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:11.929532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.929422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b65941-f0b7-4951-a28a-2f617da8b32e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:11.929532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.929510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:11.929612 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:11.929565    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbgln\" (UniqueName: \"kubernetes.io/projected/c7b65941-f0b7-4951-a28a-2f617da8b32e-kube-api-access-lbgln\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.030470 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.030429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b65941-f0b7-4951-a28a-2f617da8b32e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.030656 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.030503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.030656 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.030560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgln\" (UniqueName: \"kubernetes.io/projected/c7b65941-f0b7-4951-a28a-2f617da8b32e-kube-api-access-lbgln\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.030656 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:12.030598    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-serving-cert: secret "maas-api-e2e-worker-w4-7025b9-serving-cert" not found
Sep 10 23:14:12.030656 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.030612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.030963 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:12.030673    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls podName:c7b65941-f0b7-4951-a28a-2f617da8b32e nodeName:}" failed. No retries permitted until 2026-09-10 23:14:12.530656419 +0000 UTC m=+756.816677597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls") pod "maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" (UID: "c7b65941-f0b7-4951-a28a-2f617da8b32e") : secret "maas-api-e2e-worker-w4-7025b9-serving-cert" not found
Sep 10 23:14:12.031311 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.031281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b65941-f0b7-4951-a28a-2f617da8b32e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.033336 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.033315    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-metrics-certs\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.038861 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.038838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbgln\" (UniqueName: \"kubernetes.io/projected/c7b65941-f0b7-4951-a28a-2f617da8b32e-kube-api-access-lbgln\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.535576 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.535546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.537848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.537826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.769788 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.769753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:12.887441 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.887414    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"]
Sep 10 23:14:12.891780 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.891754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:12.893989 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.893965    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-679c94-serving-cert\""
Sep 10 23:14:12.902606 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:12.902580    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"]
Sep 10 23:14:13.039821 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.039788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-maas-api-tls\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.039998 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.039893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-metrics-certs\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.039998 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.039980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.039998 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.039998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq4wg\" (UniqueName: \"kubernetes.io/projected/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-kube-api-access-dq4wg\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.042077 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.042052    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"]
Sep 10 23:14:13.043611 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:14:13.043588    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b65941_f0b7_4951_a28a_2f617da8b32e.slice/crio-5fcf24dd8ed6621690ff9ff7e63fa2fe32f23e2e969a9b487295d12bc2b6b48f WatchSource:0}: Error finding container 5fcf24dd8ed6621690ff9ff7e63fa2fe32f23e2e969a9b487295d12bc2b6b48f: Status 404 returned error can't find the container with id 5fcf24dd8ed6621690ff9ff7e63fa2fe32f23e2e969a9b487295d12bc2b6b48f
Sep 10 23:14:13.140574 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.140512    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-maas-api-tls\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.140683 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.140574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-metrics-certs\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.140683 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.140620    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.140683 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.140636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq4wg\" (UniqueName: \"kubernetes.io/projected/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-kube-api-access-dq4wg\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.141343 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.141319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.143047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.143026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-maas-api-tls\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.143127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.143066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-metrics-certs\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.150190 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.150169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq4wg\" (UniqueName: \"kubernetes.io/projected/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-kube-api-access-dq4wg\") pod \"maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.204127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.204100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:13.479025 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.478994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"]
Sep 10 23:14:13.482739 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:14:13.479945    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20922ff7_5cda_46b6_bd9b_eb7fc9e9873c.slice/crio-cce52eccbb78ac117251d3bc81bd4fc729329dc1d9b7b590cdac8c6fea14cc39 WatchSource:0}: Error finding container cce52eccbb78ac117251d3bc81bd4fc729329dc1d9b7b590cdac8c6fea14cc39: Status 404 returned error can't find the container with id cce52eccbb78ac117251d3bc81bd4fc729329dc1d9b7b590cdac8c6fea14cc39
Sep 10 23:14:13.566013 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.565982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4" event={"ID":"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c","Type":"ContainerStarted","Data":"cce52eccbb78ac117251d3bc81bd4fc729329dc1d9b7b590cdac8c6fea14cc39"}
Sep 10 23:14:13.567427 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:13.567401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" event={"ID":"c7b65941-f0b7-4951-a28a-2f617da8b32e","Type":"ContainerStarted","Data":"5fcf24dd8ed6621690ff9ff7e63fa2fe32f23e2e969a9b487295d12bc2b6b48f"}
Sep 10 23:14:14.056635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.056605    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"]
Sep 10 23:14:14.060628 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.060607    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.063138 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.063113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6ce17e-serving-cert\""
Sep 10 23:14:14.072377 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.072353    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"]
Sep 10 23:14:14.150835 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.150811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.150964 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.150866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79hk\" (UniqueName: \"kubernetes.io/projected/84ca9abb-323e-4bf4-a63b-1ee5232a0364-kube-api-access-h79hk\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.150964 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.150944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ca9abb-323e-4bf4-a63b-1ee5232a0364-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.151036 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.150977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.251982 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.251897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.251982 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.251970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.252199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.252065    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79hk\" (UniqueName: \"kubernetes.io/projected/84ca9abb-323e-4bf4-a63b-1ee5232a0364-kube-api-access-h79hk\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.252199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.252108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ca9abb-323e-4bf4-a63b-1ee5232a0364-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.252199 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:14.252114    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-serving-cert: secret "maas-api-e2e-worker-w0-6ce17e-serving-cert" not found
Sep 10 23:14:14.252199 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:14.252174    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls podName:84ca9abb-323e-4bf4-a63b-1ee5232a0364 nodeName:}" failed. No retries permitted until 2026-09-10 23:14:14.752159315 +0000 UTC m=+759.038180497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls") pod "maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" (UID: "84ca9abb-323e-4bf4-a63b-1ee5232a0364") : secret "maas-api-e2e-worker-w0-6ce17e-serving-cert" not found
Sep 10 23:14:14.252749 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.252729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ca9abb-323e-4bf4-a63b-1ee5232a0364-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.254282 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.254262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.261521 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.261497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79hk\" (UniqueName: \"kubernetes.io/projected/84ca9abb-323e-4bf4-a63b-1ee5232a0364-kube-api-access-h79hk\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.573754 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.573699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4" event={"ID":"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c","Type":"ContainerStarted","Data":"165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763"}
Sep 10 23:14:14.573995 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.573959    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:14.575304 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.575282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" event={"ID":"c7b65941-f0b7-4951-a28a-2f617da8b32e","Type":"ContainerStarted","Data":"0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f"}
Sep 10 23:14:14.575436 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.575416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:14.596264 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.596218    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4" podStartSLOduration=2.143484535 podStartE2EDuration="2.596204803s" totalImagesPullingTime="452.720268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:14.595995643 +0000 UTC m=+758.882016844" watchObservedRunningTime="2026-09-10 23:14:14.596204803 +0000 UTC m=+758.882225998"
Sep 10 23:14:14.618020 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.617979    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" podStartSLOduration=3.093971707 podStartE2EDuration="3.6179651s" totalImagesPullingTime="523.993393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:14.616190119 +0000 UTC m=+758.902211320" watchObservedRunningTime="2026-09-10 23:14:14.6179651 +0000 UTC m=+758.903986303"
Sep 10 23:14:14.756765 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.756733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.759149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.759125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:14.973629 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:14.973602    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:15.251157 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:15.251134    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"]
Sep 10 23:14:15.253168 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:14:15.253140    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ca9abb_323e_4bf4_a63b_1ee5232a0364.slice/crio-c113acf68ff8ca338625c1bdfa1c5d68e7846b8371352a4961c0807283feeb9f WatchSource:0}: Error finding container c113acf68ff8ca338625c1bdfa1c5d68e7846b8371352a4961c0807283feeb9f: Status 404 returned error can't find the container with id c113acf68ff8ca338625c1bdfa1c5d68e7846b8371352a4961c0807283feeb9f
Sep 10 23:14:15.581459 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:15.581376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" event={"ID":"84ca9abb-323e-4bf4-a63b-1ee5232a0364","Type":"ContainerStarted","Data":"c113acf68ff8ca338625c1bdfa1c5d68e7846b8371352a4961c0807283feeb9f"}
Sep 10 23:14:16.589427 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:16.589390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" event={"ID":"84ca9abb-323e-4bf4-a63b-1ee5232a0364","Type":"ContainerStarted","Data":"405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045"}
Sep 10 23:14:16.589820 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:16.589463    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:16.612196 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:16.612145    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" podStartSLOduration=2.212751443 podStartE2EDuration="2.612129333s" totalImagesPullingTime="399.37789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:16.610755796 +0000 UTC m=+760.896776996" watchObservedRunningTime="2026-09-10 23:14:16.612129333 +0000 UTC m=+760.898150585"
Sep 10 23:14:19.797897 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.797865    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"]
Sep 10 23:14:19.801897 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.801872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:19.804372 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.804347    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-689fe3-serving-cert\""
Sep 10 23:14:19.812812 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.812763    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"]
Sep 10 23:14:19.902586 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.902560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-metrics-certs\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:19.902586 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.902589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpktf\" (UniqueName: \"kubernetes.io/projected/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-kube-api-access-hpktf\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:19.902795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.902731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:19.902830 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:19.902806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-openshift-service-ca\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.003519 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.003482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-openshift-service-ca\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.003657 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.003526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-metrics-certs\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.003657 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.003548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpktf\" (UniqueName: \"kubernetes.io/projected/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-kube-api-access-hpktf\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.003657 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.003640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.003823 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:20.003801    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-serving-cert: secret "maas-api-e2e-autores-689fe3-serving-cert" not found
Sep 10 23:14:20.003874 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:20.003864    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls podName:528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea nodeName:}" failed. No retries permitted until 2026-09-10 23:14:20.503849378 +0000 UTC m=+764.789870563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls") pod "maas-api-e2e-autores-689fe3-85dff97866-djmmt" (UID: "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea") : secret "maas-api-e2e-autores-689fe3-serving-cert" not found
Sep 10 23:14:20.004225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.004203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-openshift-service-ca\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.006019 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.006000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-metrics-certs\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.017726 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.017685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpktf\" (UniqueName: \"kubernetes.io/projected/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-kube-api-access-hpktf\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.508050 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.508008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.510440 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.510416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls\") pod \"maas-api-e2e-autores-689fe3-85dff97866-djmmt\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.586936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.586911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:14:20.587358 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.587337    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:14:20.714541 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.714513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:20.997257 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:20.997208    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"]
Sep 10 23:14:20.999903 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:14:20.999873    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528f31f7_ab9b_4cf1_b86b_16c6fa49a2ea.slice/crio-6e122cef2ef894ee9dc6ec945d33e9c4b46e3b2f2c814119cf7c92af36bb85ce WatchSource:0}: Error finding container 6e122cef2ef894ee9dc6ec945d33e9c4b46e3b2f2c814119cf7c92af36bb85ce: Status 404 returned error can't find the container with id 6e122cef2ef894ee9dc6ec945d33e9c4b46e3b2f2c814119cf7c92af36bb85ce
Sep 10 23:14:21.618131 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:21.618052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt" event={"ID":"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea","Type":"ContainerStarted","Data":"f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0"}
Sep 10 23:14:21.618131 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:21.618116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt" event={"ID":"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea","Type":"ContainerStarted","Data":"6e122cef2ef894ee9dc6ec945d33e9c4b46e3b2f2c814119cf7c92af36bb85ce"}
Sep 10 23:14:21.618343 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:21.618161    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:21.637013 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:21.636962    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt" podStartSLOduration=2.299644704 podStartE2EDuration="2.636942695s" totalImagesPullingTime="337.297991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:21.636725354 +0000 UTC m=+765.922746557" watchObservedRunningTime="2026-09-10 23:14:21.636942695 +0000 UTC m=+765.922963896"
Sep 10 23:14:22.600546 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:22.600499    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:14:27.627925 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:27.627895    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:14:42.005824 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.005791    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"]
Sep 10 23:14:42.009328 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.009308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.011615 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.011593    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-568856-serving-cert\""
Sep 10 23:14:42.019696 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.019669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"]
Sep 10 23:14:42.087889 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.087857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d455f2e5-9f51-4947-af25-b4ced04337fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.088065 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.087905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-metrics-certs\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.088065 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.087934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.088065 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.088041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjb7\" (UniqueName: \"kubernetes.io/projected/d455f2e5-9f51-4947-af25-b4ced04337fe-kube-api-access-wqjb7\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.188633 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.188599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d455f2e5-9f51-4947-af25-b4ced04337fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.188633 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.188641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-metrics-certs\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.188914 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.188661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.188914 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.188736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjb7\" (UniqueName: \"kubernetes.io/projected/d455f2e5-9f51-4947-af25-b4ced04337fe-kube-api-access-wqjb7\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.188914 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:42.188840    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-serving-cert: secret "maas-api-e2e-inf-b-568856-serving-cert" not found
Sep 10 23:14:42.189078 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:42.188923    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls podName:d455f2e5-9f51-4947-af25-b4ced04337fe nodeName:}" failed. No retries permitted until 2026-09-10 23:14:42.688905887 +0000 UTC m=+786.974927066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls") pod "maas-api-e2e-inf-b-568856-578dd574d6-277sq" (UID: "d455f2e5-9f51-4947-af25-b4ced04337fe") : secret "maas-api-e2e-inf-b-568856-serving-cert" not found
Sep 10 23:14:42.189366 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.189343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d455f2e5-9f51-4947-af25-b4ced04337fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.191143 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.191125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-metrics-certs\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.197729 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.197689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjb7\" (UniqueName: \"kubernetes.io/projected/d455f2e5-9f51-4947-af25-b4ced04337fe-kube-api-access-wqjb7\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.692462 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.692433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:42.692766 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:42.692593    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-serving-cert: secret "maas-api-e2e-inf-b-568856-serving-cert" not found
Sep 10 23:14:42.692766 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:14:42.692664    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls podName:d455f2e5-9f51-4947-af25-b4ced04337fe nodeName:}" failed. No retries permitted until 2026-09-10 23:14:43.692645781 +0000 UTC m=+787.978666979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls") pod "maas-api-e2e-inf-b-568856-578dd574d6-277sq" (UID: "d455f2e5-9f51-4947-af25-b4ced04337fe") : secret "maas-api-e2e-inf-b-568856-serving-cert" not found
Sep 10 23:14:42.787793 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.787763    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"]
Sep 10 23:14:42.791165 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.791144    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:42.793596 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.793574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-568856-dockercfg-4sgps\""
Sep 10 23:14:42.793738 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.793628    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-568856\""
Sep 10 23:14:42.799826 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.799800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"]
Sep 10 23:14:42.894490 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.894457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:42.894654 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.894513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpchk\" (UniqueName: \"kubernetes.io/projected/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-kube-api-access-tpchk\") pod \"payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:42.995819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.995733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpchk\" (UniqueName: \"kubernetes.io/projected/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-kube-api-access-tpchk\") pod \"payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:42.996000 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.995933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:42.996822 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:42.996795    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:43.008119 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.008094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpchk\" (UniqueName: \"kubernetes.io/projected/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-kube-api-access-tpchk\") pod \"payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:43.105223 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.105185    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:43.379006 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.378980    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"]
Sep 10 23:14:43.380893 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:14:43.380860    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c0b48e_fcfb_4164_b4f6_f29a04b7348d.slice/crio-b091b2aa5a44b5456c8c05b3150fbde7d9708b2c4de98bac3ddcbda5a60f7678 WatchSource:0}: Error finding container b091b2aa5a44b5456c8c05b3150fbde7d9708b2c4de98bac3ddcbda5a60f7678: Status 404 returned error can't find the container with id b091b2aa5a44b5456c8c05b3150fbde7d9708b2c4de98bac3ddcbda5a60f7678
Sep 10 23:14:43.702271 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.702233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:43.705189 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.705160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-568856-578dd574d6-277sq\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:43.726256 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.726221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g" event={"ID":"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d","Type":"ContainerStarted","Data":"3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264"}
Sep 10 23:14:43.726388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.726271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g" event={"ID":"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d","Type":"ContainerStarted","Data":"b091b2aa5a44b5456c8c05b3150fbde7d9708b2c4de98bac3ddcbda5a60f7678"}
Sep 10 23:14:43.726388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.726348    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:14:43.744159 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.744112    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g" podStartSLOduration=1.7440974 podStartE2EDuration="1.7440974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:43.742762921 +0000 UTC m=+788.028784122" watchObservedRunningTime="2026-09-10 23:14:43.7440974 +0000 UTC m=+788.030118651"
Sep 10 23:14:43.821028 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:43.820997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:44.099100 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:44.099075    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"]
Sep 10 23:14:44.101298 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:14:44.101269    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd455f2e5_9f51_4947_af25_b4ced04337fe.slice/crio-4799f154087a5921dc5f144eb41b809034e2820278ab3da23b13eb74372cfc4a WatchSource:0}: Error finding container 4799f154087a5921dc5f144eb41b809034e2820278ab3da23b13eb74372cfc4a: Status 404 returned error can't find the container with id 4799f154087a5921dc5f144eb41b809034e2820278ab3da23b13eb74372cfc4a
Sep 10 23:14:44.733640 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:44.733600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq" event={"ID":"d455f2e5-9f51-4947-af25-b4ced04337fe","Type":"ContainerStarted","Data":"482bb68f232ce442daedddc1f977a6aaf340d8f47e44a7d5553ef020101ca10d"}
Sep 10 23:14:44.733838 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:44.733691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:44.733838 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:44.733752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq" event={"ID":"d455f2e5-9f51-4947-af25-b4ced04337fe","Type":"ContainerStarted","Data":"4799f154087a5921dc5f144eb41b809034e2820278ab3da23b13eb74372cfc4a"}
Sep 10 23:14:44.752876 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:44.752827    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq" podStartSLOduration=3.320220252 podStartE2EDuration="3.752813433s" totalImagesPullingTime="432.593181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:44.750595434 +0000 UTC m=+789.036616648" watchObservedRunningTime="2026-09-10 23:14:44.752813433 +0000 UTC m=+789.038834684"
Sep 10 23:14:50.744371 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:50.744342    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:14:54.735695 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:14:54.735662    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:16:19.156617 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.156575    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"]
Sep 10 23:16:19.160533 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.160511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.162628 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.162608    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-51383f0b\""
Sep 10 23:16:19.162730 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.162608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-51383f0b-dockercfg-pwnrc\""
Sep 10 23:16:19.170022 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.169992    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"]
Sep 10 23:16:19.323697 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.323649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cdb5de-de4b-452c-80ad-321e590043de-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.323903 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.323769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8clj6\" (UniqueName: \"kubernetes.io/projected/e2cdb5de-de4b-452c-80ad-321e590043de-kube-api-access-8clj6\") pod \"payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.425090 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.425061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clj6\" (UniqueName: \"kubernetes.io/projected/e2cdb5de-de4b-452c-80ad-321e590043de-kube-api-access-8clj6\") pod \"payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.425230 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.425149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cdb5de-de4b-452c-80ad-321e590043de-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.425803 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.425784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cdb5de-de4b-452c-80ad-321e590043de-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.432556 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.432535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clj6\" (UniqueName: \"kubernetes.io/projected/e2cdb5de-de4b-452c-80ad-321e590043de-kube-api-access-8clj6\") pod \"payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.472364 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.472342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:19.744379 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:19.744356    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"]
Sep 10 23:16:19.746598 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:16:19.746569    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cdb5de_de4b_452c_80ad_321e590043de.slice/crio-f4538dd061e6df27a2d590c4d35cd4903e42866496b45c2ab24c990d549c8fd6 WatchSource:0}: Error finding container f4538dd061e6df27a2d590c4d35cd4903e42866496b45c2ab24c990d549c8fd6: Status 404 returned error can't find the container with id f4538dd061e6df27a2d590c4d35cd4903e42866496b45c2ab24c990d549c8fd6
Sep 10 23:16:20.216246 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:20.216212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656" event={"ID":"e2cdb5de-de4b-452c-80ad-321e590043de","Type":"ContainerStarted","Data":"98d0e65f49cd9bb95fb5d6b88016899bc6c27c98ce8305f92f708906fecdc164"}
Sep 10 23:16:20.216802 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:20.216259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656" event={"ID":"e2cdb5de-de4b-452c-80ad-321e590043de","Type":"ContainerStarted","Data":"f4538dd061e6df27a2d590c4d35cd4903e42866496b45c2ab24c990d549c8fd6"}
Sep 10 23:16:20.216802 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:20.216291    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:20.232692 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:20.232642    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656" podStartSLOduration=1.2326273859999999 podStartE2EDuration="1.232627386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:20.230901104 +0000 UTC m=+884.516922305" watchObservedRunningTime="2026-09-10 23:16:20.232627386 +0000 UTC m=+884.518648586"
Sep 10 23:16:31.223838 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:31.223759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:36.151836 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:36.151808    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:16:36.152260 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:36.152150    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:16:47.846832 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:47.846795    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"]
Sep 10 23:16:47.849352 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:47.847019    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt" podUID="528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" containerName="maas-api" containerID="cri-o://f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0" gracePeriod=30
Sep 10 23:16:48.097047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.096996    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:16:48.166675 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.166648    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls\") pod \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") "
Sep 10 23:16:48.166862 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.166688    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-metrics-certs\") pod \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") "
Sep 10 23:16:48.166862 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.166741    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpktf\" (UniqueName: \"kubernetes.io/projected/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-kube-api-access-hpktf\") pod \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") "
Sep 10 23:16:48.166862 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.166818    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-openshift-service-ca\") pod \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\" (UID: \"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea\") "
Sep 10 23:16:48.167225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.167190    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" (UID: "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:48.168909 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.168886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" (UID: "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:48.169046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.168916    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-kube-api-access-hpktf" (OuterVolumeSpecName: "kube-api-access-hpktf") pod "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" (UID: "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea"). InnerVolumeSpecName "kube-api-access-hpktf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:48.169046 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.168963    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" (UID: "528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:48.268410 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.268347    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.268410 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.268384    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.268659 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.268421    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpktf\" (UniqueName: \"kubernetes.io/projected/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-kube-api-access-hpktf\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.268659 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.268439    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.356042 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.355971    2564 generic.go:356] "Generic (PLEG): container finished" podID="528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" containerID="f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0" exitCode=0
Sep 10 23:16:48.356042 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.356015    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"
Sep 10 23:16:48.356194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.356051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt" event={"ID":"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea","Type":"ContainerDied","Data":"f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0"}
Sep 10 23:16:48.356194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.356109    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt" event={"ID":"528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea","Type":"ContainerDied","Data":"6e122cef2ef894ee9dc6ec945d33e9c4b46e3b2f2c814119cf7c92af36bb85ce"}
Sep 10 23:16:48.356194 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.356154    2564 scope.go:117] "RemoveContainer" containerID="f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0"
Sep 10 23:16:48.365088 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.365070    2564 scope.go:117] "RemoveContainer" containerID="f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0"
Sep 10 23:16:48.365356 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:16:48.365336    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0\": container with ID starting with f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0 not found: ID does not exist" containerID="f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0"
Sep 10 23:16:48.365403 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.365363    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0"} err="failed to get container status \"f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0\": rpc error: code = NotFound desc = could not find container \"f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0\": container with ID starting with f233d7889aa02e8121d1881aa1f9cadc57c36caf05c66eb4cc9b855c4389c1e0 not found: ID does not exist"
Sep 10 23:16:48.372062 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.372042    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"]
Sep 10 23:16:48.376019 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:48.376000    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-689fe3-85dff97866-djmmt"]
Sep 10 23:16:49.166199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.166171    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"]
Sep 10 23:16:49.166626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.166366    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656" podUID="e2cdb5de-de4b-452c-80ad-321e590043de" containerName="payload-pre-processing" containerID="cri-o://98d0e65f49cd9bb95fb5d6b88016899bc6c27c98ce8305f92f708906fecdc164" gracePeriod=30
Sep 10 23:16:49.364989 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.364950    2564 generic.go:356] "Generic (PLEG): container finished" podID="e2cdb5de-de4b-452c-80ad-321e590043de" containerID="98d0e65f49cd9bb95fb5d6b88016899bc6c27c98ce8305f92f708906fecdc164" exitCode=0
Sep 10 23:16:49.365140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.365028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656" event={"ID":"e2cdb5de-de4b-452c-80ad-321e590043de","Type":"ContainerDied","Data":"98d0e65f49cd9bb95fb5d6b88016899bc6c27c98ce8305f92f708906fecdc164"}
Sep 10 23:16:49.434053 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.434014    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:49.477463 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.477436    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8clj6\" (UniqueName: \"kubernetes.io/projected/e2cdb5de-de4b-452c-80ad-321e590043de-kube-api-access-8clj6\") pod \"e2cdb5de-de4b-452c-80ad-321e590043de\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") "
Sep 10 23:16:49.477603 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.477513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cdb5de-de4b-452c-80ad-321e590043de-plugins-config-volume\") pod \"e2cdb5de-de4b-452c-80ad-321e590043de\" (UID: \"e2cdb5de-de4b-452c-80ad-321e590043de\") "
Sep 10 23:16:49.477907 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.477880    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cdb5de-de4b-452c-80ad-321e590043de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2cdb5de-de4b-452c-80ad-321e590043de" (UID: "e2cdb5de-de4b-452c-80ad-321e590043de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:49.480473 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.480449    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cdb5de-de4b-452c-80ad-321e590043de-kube-api-access-8clj6" (OuterVolumeSpecName: "kube-api-access-8clj6") pod "e2cdb5de-de4b-452c-80ad-321e590043de" (UID: "e2cdb5de-de4b-452c-80ad-321e590043de"). InnerVolumeSpecName "kube-api-access-8clj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:49.578848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.578818    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8clj6\" (UniqueName: \"kubernetes.io/projected/e2cdb5de-de4b-452c-80ad-321e590043de-kube-api-access-8clj6\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:16:49.578848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:49.578844    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cdb5de-de4b-452c-80ad-321e590043de-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:16:50.198496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:50.198461    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" path="/var/lib/kubelet/pods/528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea/volumes"
Sep 10 23:16:50.372652 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:50.372624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656" event={"ID":"e2cdb5de-de4b-452c-80ad-321e590043de","Type":"ContainerDied","Data":"f4538dd061e6df27a2d590c4d35cd4903e42866496b45c2ab24c990d549c8fd6"}
Sep 10 23:16:50.372793 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:50.372669    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"
Sep 10 23:16:50.372793 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:50.372683    2564 scope.go:117] "RemoveContainer" containerID="98d0e65f49cd9bb95fb5d6b88016899bc6c27c98ce8305f92f708906fecdc164"
Sep 10 23:16:50.388510 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:50.388481    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"]
Sep 10 23:16:50.391184 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:50.391160    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-51383f0b-788557fcd-kw656"]
Sep 10 23:16:52.199817 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:16:52.199788    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2cdb5de-de4b-452c-80ad-321e590043de" path="/var/lib/kubelet/pods/e2cdb5de-de4b-452c-80ad-321e590043de/volumes"
Sep 10 23:17:36.564260 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564182    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"]
Sep 10 23:17:36.564643 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564593    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cdb5de-de4b-452c-80ad-321e590043de" containerName="payload-pre-processing"
Sep 10 23:17:36.564643 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564604    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cdb5de-de4b-452c-80ad-321e590043de" containerName="payload-pre-processing"
Sep 10 23:17:36.564643 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564623    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" containerName="maas-api"
Sep 10 23:17:36.564643 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564630    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" containerName="maas-api"
Sep 10 23:17:36.564804 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564684    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2cdb5de-de4b-452c-80ad-321e590043de" containerName="payload-pre-processing"
Sep 10 23:17:36.564804 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.564693    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="528f31f7-ab9b-4cf1-b86b-16c6fa49a2ea" containerName="maas-api"
Sep 10 23:17:36.569050 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.569026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.571465 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.571444    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b3d9be-serving-cert\""
Sep 10 23:17:36.578043 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.577621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"]
Sep 10 23:17:36.665437 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.665401    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmcq\" (UniqueName: \"kubernetes.io/projected/f2b4395d-f72c-4db9-814b-719660fbdf87-kube-api-access-xvmcq\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.665771 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.665749    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-metrics-certs\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.665875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.665799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b4395d-f72c-4db9-814b-719660fbdf87-openshift-service-ca\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.665875 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.665841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.766964 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.766935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-metrics-certs\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.767092 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.766976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b4395d-f72c-4db9-814b-719660fbdf87-openshift-service-ca\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.767092 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.767014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.767092 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.767041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmcq\" (UniqueName: \"kubernetes.io/projected/f2b4395d-f72c-4db9-814b-719660fbdf87-kube-api-access-xvmcq\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.767240 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:17:36.767171    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-serving-cert: secret "maas-api-e2e-explicit-b3d9be-serving-cert" not found
Sep 10 23:17:36.767297 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:17:36.767241    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls podName:f2b4395d-f72c-4db9-814b-719660fbdf87 nodeName:}" failed. No retries permitted until 2026-09-10 23:17:37.267224801 +0000 UTC m=+961.553245980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls") pod "maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" (UID: "f2b4395d-f72c-4db9-814b-719660fbdf87") : secret "maas-api-e2e-explicit-b3d9be-serving-cert" not found
Sep 10 23:17:36.767936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.767900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b4395d-f72c-4db9-814b-719660fbdf87-openshift-service-ca\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.769586 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.769568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-metrics-certs\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:36.777161 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:36.777140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmcq\" (UniqueName: \"kubernetes.io/projected/f2b4395d-f72c-4db9-814b-719660fbdf87-kube-api-access-xvmcq\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:37.271837 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:37.271801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:37.274051 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:37.274022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls\") pod \"maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:37.482408 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:37.482377    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:37.754235 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:37.754208    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"]
Sep 10 23:17:37.755866 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:17:37.755843    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b4395d_f72c_4db9_814b_719660fbdf87.slice/crio-35fd583d54c4376e6161c514c63f4bbaed4dee6cef95889ea04405b312a14b6c WatchSource:0}: Error finding container 35fd583d54c4376e6161c514c63f4bbaed4dee6cef95889ea04405b312a14b6c: Status 404 returned error can't find the container with id 35fd583d54c4376e6161c514c63f4bbaed4dee6cef95889ea04405b312a14b6c
Sep 10 23:17:37.757174 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:37.757155    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:17:38.613905 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:38.613871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" event={"ID":"f2b4395d-f72c-4db9-814b-719660fbdf87","Type":"ContainerStarted","Data":"d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df"}
Sep 10 23:17:38.614090 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:38.613931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" event={"ID":"f2b4395d-f72c-4db9-814b-719660fbdf87","Type":"ContainerStarted","Data":"35fd583d54c4376e6161c514c63f4bbaed4dee6cef95889ea04405b312a14b6c"}
Sep 10 23:17:38.614090 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:38.613957    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:17:38.633970 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:38.633921    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" podStartSLOduration=2.230285838 podStartE2EDuration="2.633908951s" totalImagesPullingTime="403.623113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:38.632853115 +0000 UTC m=+962.918874321" watchObservedRunningTime="2026-09-10 23:17:38.633908951 +0000 UTC m=+962.919930178"
Sep 10 23:17:44.624446 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:17:44.624415    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:18:33.693037 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.693002    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"]
Sep 10 23:18:33.696623 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.696599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.698895 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.698874    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5d061fe3-dockercfg-h8fm8\""
Sep 10 23:18:33.698999 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.698921    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5d061fe3\""
Sep 10 23:18:33.708527 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.708508    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"]
Sep 10 23:18:33.712365 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.712347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"]
Sep 10 23:18:33.712444 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.712424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:33.722535 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.722508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"]
Sep 10 23:18:33.739456 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.739434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f776696-16d5-49d8-9dcc-02c352baa015-plugins-config-volume\") pod \"payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") " pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:33.739553 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.739490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbkm\" (UniqueName: \"kubernetes.io/projected/6f776696-16d5-49d8-9dcc-02c352baa015-kube-api-access-4wbkm\") pod \"payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") " pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:33.739553 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.739506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.739636 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.739600    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljjw\" (UniqueName: \"kubernetes.io/projected/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-kube-api-access-wljjw\") pod \"payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.840333 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.840304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbkm\" (UniqueName: \"kubernetes.io/projected/6f776696-16d5-49d8-9dcc-02c352baa015-kube-api-access-4wbkm\") pod \"payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") " pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:33.840333 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.840337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.840534 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.840363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljjw\" (UniqueName: \"kubernetes.io/projected/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-kube-api-access-wljjw\") pod \"payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.840534 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.840419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f776696-16d5-49d8-9dcc-02c352baa015-plugins-config-volume\") pod \"payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") " pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:33.841099 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.841079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.841242 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.841215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f776696-16d5-49d8-9dcc-02c352baa015-plugins-config-volume\") pod \"payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") " pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:33.849116 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.849084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljjw\" (UniqueName: \"kubernetes.io/projected/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-kube-api-access-wljjw\") pod \"payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:33.849241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:33.849218    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbkm\" (UniqueName: \"kubernetes.io/projected/6f776696-16d5-49d8-9dcc-02c352baa015-kube-api-access-4wbkm\") pod \"payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") " pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:34.006923 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.006829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:34.024054 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.024023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:34.283970 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.283945    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"]
Sep 10 23:18:34.285930 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:18:34.285900    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f0d93a_7ff8_4ef8_804b_343ed685a0ee.slice/crio-2320c0f203206b08a878cc75dbf4d44dc653de6bf1e1dec1017a1048982c91c4 WatchSource:0}: Error finding container 2320c0f203206b08a878cc75dbf4d44dc653de6bf1e1dec1017a1048982c91c4: Status 404 returned error can't find the container with id 2320c0f203206b08a878cc75dbf4d44dc653de6bf1e1dec1017a1048982c91c4
Sep 10 23:18:34.306873 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.306852    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"]
Sep 10 23:18:34.308542 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:18:34.308516    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f776696_16d5_49d8_9dcc_02c352baa015.slice/crio-eed6156e9489db8ae8418a90be56a2152888bcf285fd4d2f948bfbf3a469b8b6 WatchSource:0}: Error finding container eed6156e9489db8ae8418a90be56a2152888bcf285fd4d2f948bfbf3a469b8b6: Status 404 returned error can't find the container with id eed6156e9489db8ae8418a90be56a2152888bcf285fd4d2f948bfbf3a469b8b6
Sep 10 23:18:34.901532 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.901501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49" event={"ID":"21f0d93a-7ff8-4ef8-804b-343ed685a0ee","Type":"ContainerStarted","Data":"bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33"}
Sep 10 23:18:34.902002 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.901560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49" event={"ID":"21f0d93a-7ff8-4ef8-804b-343ed685a0ee","Type":"ContainerStarted","Data":"2320c0f203206b08a878cc75dbf4d44dc653de6bf1e1dec1017a1048982c91c4"}
Sep 10 23:18:34.902002 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.901699    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:34.903264 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.903242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h" event={"ID":"6f776696-16d5-49d8-9dcc-02c352baa015","Type":"ContainerStarted","Data":"4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250"}
Sep 10 23:18:34.903377 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.903282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h" event={"ID":"6f776696-16d5-49d8-9dcc-02c352baa015","Type":"ContainerStarted","Data":"eed6156e9489db8ae8418a90be56a2152888bcf285fd4d2f948bfbf3a469b8b6"}
Sep 10 23:18:34.903377 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.903308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:18:34.916668 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.916625    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49" podStartSLOduration=1.9166122429999999 podStartE2EDuration="1.916612243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:34.916167786 +0000 UTC m=+1019.202188988" watchObservedRunningTime="2026-09-10 23:18:34.916612243 +0000 UTC m=+1019.202633444"
Sep 10 23:18:34.932444 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:34.932403    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h" podStartSLOduration=1.932390688 podStartE2EDuration="1.932390688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:34.930486893 +0000 UTC m=+1019.216508094" watchObservedRunningTime="2026-09-10 23:18:34.932390688 +0000 UTC m=+1019.218411889"
Sep 10 23:18:45.909595 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:45.909552    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:18:45.910167 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:18:45.909822    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:19:14.382637 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.382587    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"]
Sep 10 23:19:14.383144 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.382867    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h" podUID="6f776696-16d5-49d8-9dcc-02c352baa015" containerName="payload-processing" containerID="cri-o://4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250" gracePeriod=30
Sep 10 23:19:14.390616 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.390587    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"]
Sep 10 23:19:14.390888 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.390827    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49" podUID="21f0d93a-7ff8-4ef8-804b-343ed685a0ee" containerName="payload-pre-processing" containerID="cri-o://bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33" gracePeriod=30
Sep 10 23:19:14.662268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.662248    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:19:14.676349 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.676324    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:19:14.787792 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.787761    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wljjw\" (UniqueName: \"kubernetes.io/projected/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-kube-api-access-wljjw\") pod \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") "
Sep 10 23:19:14.787966 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.787844    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f776696-16d5-49d8-9dcc-02c352baa015-plugins-config-volume\") pod \"6f776696-16d5-49d8-9dcc-02c352baa015\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") "
Sep 10 23:19:14.787966 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.787941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-plugins-config-volume\") pod \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\" (UID: \"21f0d93a-7ff8-4ef8-804b-343ed685a0ee\") "
Sep 10 23:19:14.788089 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.787996    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wbkm\" (UniqueName: \"kubernetes.io/projected/6f776696-16d5-49d8-9dcc-02c352baa015-kube-api-access-4wbkm\") pod \"6f776696-16d5-49d8-9dcc-02c352baa015\" (UID: \"6f776696-16d5-49d8-9dcc-02c352baa015\") "
Sep 10 23:19:14.788267 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.788243    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f776696-16d5-49d8-9dcc-02c352baa015-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f776696-16d5-49d8-9dcc-02c352baa015" (UID: "6f776696-16d5-49d8-9dcc-02c352baa015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:14.788344 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.788309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21f0d93a-7ff8-4ef8-804b-343ed685a0ee" (UID: "21f0d93a-7ff8-4ef8-804b-343ed685a0ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:14.789908 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.789886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-kube-api-access-wljjw" (OuterVolumeSpecName: "kube-api-access-wljjw") pod "21f0d93a-7ff8-4ef8-804b-343ed685a0ee" (UID: "21f0d93a-7ff8-4ef8-804b-343ed685a0ee"). InnerVolumeSpecName "kube-api-access-wljjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:14.789968 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.789903    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f776696-16d5-49d8-9dcc-02c352baa015-kube-api-access-4wbkm" (OuterVolumeSpecName: "kube-api-access-4wbkm") pod "6f776696-16d5-49d8-9dcc-02c352baa015" (UID: "6f776696-16d5-49d8-9dcc-02c352baa015"). InnerVolumeSpecName "kube-api-access-4wbkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:14.889626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.889601    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.889626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.889623    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbkm\" (UniqueName: \"kubernetes.io/projected/6f776696-16d5-49d8-9dcc-02c352baa015-kube-api-access-4wbkm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.889626 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.889632    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljjw\" (UniqueName: \"kubernetes.io/projected/21f0d93a-7ff8-4ef8-804b-343ed685a0ee-kube-api-access-wljjw\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:14.889860 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:14.889640    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f776696-16d5-49d8-9dcc-02c352baa015-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:15.106294 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.106259    2564 generic.go:356] "Generic (PLEG): container finished" podID="21f0d93a-7ff8-4ef8-804b-343ed685a0ee" containerID="bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33" exitCode=0
Sep 10 23:19:15.106468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.106302    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"
Sep 10 23:19:15.106468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.106353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49" event={"ID":"21f0d93a-7ff8-4ef8-804b-343ed685a0ee","Type":"ContainerDied","Data":"bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33"}
Sep 10 23:19:15.106468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.106400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49" event={"ID":"21f0d93a-7ff8-4ef8-804b-343ed685a0ee","Type":"ContainerDied","Data":"2320c0f203206b08a878cc75dbf4d44dc653de6bf1e1dec1017a1048982c91c4"}
Sep 10 23:19:15.106468 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.106427    2564 scope.go:117] "RemoveContainer" containerID="bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33"
Sep 10 23:19:15.107825 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.107803    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f776696-16d5-49d8-9dcc-02c352baa015" containerID="4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250" exitCode=0
Sep 10 23:19:15.107922 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.107846    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"
Sep 10 23:19:15.107922 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.107885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h" event={"ID":"6f776696-16d5-49d8-9dcc-02c352baa015","Type":"ContainerDied","Data":"4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250"}
Sep 10 23:19:15.108002 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.107938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h" event={"ID":"6f776696-16d5-49d8-9dcc-02c352baa015","Type":"ContainerDied","Data":"eed6156e9489db8ae8418a90be56a2152888bcf285fd4d2f948bfbf3a469b8b6"}
Sep 10 23:19:15.115656 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.115533    2564 scope.go:117] "RemoveContainer" containerID="bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33"
Sep 10 23:19:15.115950 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:19:15.115924    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33\": container with ID starting with bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33 not found: ID does not exist" containerID="bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33"
Sep 10 23:19:15.116041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.115956    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33"} err="failed to get container status \"bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33\": rpc error: code = NotFound desc = could not find container \"bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33\": container with ID starting with bda22338764c8c090319abb7971bd81f945461c036ddfd8ebc1746460d86eb33 not found: ID does not exist"
Sep 10 23:19:15.116041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.115978    2564 scope.go:117] "RemoveContainer" containerID="4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250"
Sep 10 23:19:15.123793 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.123776    2564 scope.go:117] "RemoveContainer" containerID="4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250"
Sep 10 23:19:15.124028 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:19:15.124010    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250\": container with ID starting with 4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250 not found: ID does not exist" containerID="4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250"
Sep 10 23:19:15.124076 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.124035    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250"} err="failed to get container status \"4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250\": rpc error: code = NotFound desc = could not find container \"4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250\": container with ID starting with 4c8f3a88d695a61fedcd3e7583a976b2d1256b9ccfdfa3bf9564d06ee4cfb250 not found: ID does not exist"
Sep 10 23:19:15.130610 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.130589    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"]
Sep 10 23:19:15.132568 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.132550    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5d061fe3-7cbcb9d494-plm49"]
Sep 10 23:19:15.141586 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.141563    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"]
Sep 10 23:19:15.148650 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:15.148631    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5d061fe3-95ccb6b8f-9sl4h"]
Sep 10 23:19:16.201758 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:16.201725    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f0d93a-7ff8-4ef8-804b-343ed685a0ee" path="/var/lib/kubelet/pods/21f0d93a-7ff8-4ef8-804b-343ed685a0ee/volumes"
Sep 10 23:19:16.202222 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:16.202207    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f776696-16d5-49d8-9dcc-02c352baa015" path="/var/lib/kubelet/pods/6f776696-16d5-49d8-9dcc-02c352baa015/volumes"
Sep 10 23:19:39.768974 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.768939    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"]
Sep 10 23:19:39.769351 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.769336    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f0d93a-7ff8-4ef8-804b-343ed685a0ee" containerName="payload-pre-processing"
Sep 10 23:19:39.769351 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.769346    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f0d93a-7ff8-4ef8-804b-343ed685a0ee" containerName="payload-pre-processing"
Sep 10 23:19:39.769433 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.769366    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f776696-16d5-49d8-9dcc-02c352baa015" containerName="payload-processing"
Sep 10 23:19:39.769433 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.769375    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f776696-16d5-49d8-9dcc-02c352baa015" containerName="payload-processing"
Sep 10 23:19:39.769498 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.769463    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f776696-16d5-49d8-9dcc-02c352baa015" containerName="payload-processing"
Sep 10 23:19:39.769498 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.769473    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f0d93a-7ff8-4ef8-804b-343ed685a0ee" containerName="payload-pre-processing"
Sep 10 23:19:39.774022 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.773994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:39.776369 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.776349    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bda598\""
Sep 10 23:19:39.776499 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.776371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bda598-dockercfg-bws45\""
Sep 10 23:19:39.779927 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.779904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"]
Sep 10 23:19:39.801829 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.801807    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"]
Sep 10 23:19:39.805138 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.805123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:39.815456 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.815435    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"]
Sep 10 23:19:39.911171 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.911136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:39.911405 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.911234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcs4\" (UniqueName: \"kubernetes.io/projected/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-kube-api-access-xjcs4\") pod \"payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:39.911405 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.911268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d08644-a56b-4a00-83bd-d3f98bda474e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:39.911513 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:39.911428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hldm\" (UniqueName: \"kubernetes.io/projected/f5d08644-a56b-4a00-83bd-d3f98bda474e-kube-api-access-6hldm\") pod \"payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:40.011989 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.011957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d08644-a56b-4a00-83bd-d3f98bda474e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:40.012150 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.012030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hldm\" (UniqueName: \"kubernetes.io/projected/f5d08644-a56b-4a00-83bd-d3f98bda474e-kube-api-access-6hldm\") pod \"payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:40.012150 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.012109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:40.012150 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.012135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcs4\" (UniqueName: \"kubernetes.io/projected/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-kube-api-access-xjcs4\") pod \"payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:40.012756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.012732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d08644-a56b-4a00-83bd-d3f98bda474e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:40.012839 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.012822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:40.019971 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.019920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcs4\" (UniqueName: \"kubernetes.io/projected/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-kube-api-access-xjcs4\") pod \"payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:40.020045 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.019984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hldm\" (UniqueName: \"kubernetes.io/projected/f5d08644-a56b-4a00-83bd-d3f98bda474e-kube-api-access-6hldm\") pod \"payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:40.086418 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.086380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:40.115557 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.115533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:40.360726 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.360606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"]
Sep 10 23:19:40.362789 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:19:40.362762    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d08644_a56b_4a00_83bd_d3f98bda474e.slice/crio-dac28e9cf405b3d6572e42aa799703176f6b011899277705fbd0dc14582b1e46 WatchSource:0}: Error finding container dac28e9cf405b3d6572e42aa799703176f6b011899277705fbd0dc14582b1e46: Status 404 returned error can't find the container with id dac28e9cf405b3d6572e42aa799703176f6b011899277705fbd0dc14582b1e46
Sep 10 23:19:40.397700 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:40.397625    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"]
Sep 10 23:19:40.399920 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:19:40.399897    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9463fd1f_3d51_48e3_9c2b_fd75e7f8915c.slice/crio-8854153cf8ea8c6766268a6ee695a60dca175d278b1c3d57372f7e8ada252769 WatchSource:0}: Error finding container 8854153cf8ea8c6766268a6ee695a60dca175d278b1c3d57372f7e8ada252769: Status 404 returned error can't find the container with id 8854153cf8ea8c6766268a6ee695a60dca175d278b1c3d57372f7e8ada252769
Sep 10 23:19:41.244570 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.244522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb" event={"ID":"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c","Type":"ContainerStarted","Data":"8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8"}
Sep 10 23:19:41.245041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.244627    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:19:41.245041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.244657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb" event={"ID":"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c","Type":"ContainerStarted","Data":"8854153cf8ea8c6766268a6ee695a60dca175d278b1c3d57372f7e8ada252769"}
Sep 10 23:19:41.246177 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.246150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6" event={"ID":"f5d08644-a56b-4a00-83bd-d3f98bda474e","Type":"ContainerStarted","Data":"d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a"}
Sep 10 23:19:41.246289 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.246192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6" event={"ID":"f5d08644-a56b-4a00-83bd-d3f98bda474e","Type":"ContainerStarted","Data":"dac28e9cf405b3d6572e42aa799703176f6b011899277705fbd0dc14582b1e46"}
Sep 10 23:19:41.246289 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.246211    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:41.261441 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.261393    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb" podStartSLOduration=2.261375056 podStartE2EDuration="2.261375056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:19:41.259849691 +0000 UTC m=+1085.545870893" watchObservedRunningTime="2026-09-10 23:19:41.261375056 +0000 UTC m=+1085.547396260"
Sep 10 23:19:41.274025 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:41.273987    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6" podStartSLOduration=2.273972336 podStartE2EDuration="2.273972336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:19:41.273496093 +0000 UTC m=+1085.559517296" watchObservedRunningTime="2026-09-10 23:19:41.273972336 +0000 UTC m=+1085.559993538"
Sep 10 23:19:48.538589 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.538553    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"]
Sep 10 23:19:48.539068 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.538822    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" podUID="f2b4395d-f72c-4db9-814b-719660fbdf87" containerName="maas-api" containerID="cri-o://d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df" gracePeriod=30
Sep 10 23:19:48.793070 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.792999    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:19:48.886830 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.886795    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-metrics-certs\") pod \"f2b4395d-f72c-4db9-814b-719660fbdf87\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") "
Sep 10 23:19:48.887009 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.886863    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvmcq\" (UniqueName: \"kubernetes.io/projected/f2b4395d-f72c-4db9-814b-719660fbdf87-kube-api-access-xvmcq\") pod \"f2b4395d-f72c-4db9-814b-719660fbdf87\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") "
Sep 10 23:19:48.887009 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.886922    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls\") pod \"f2b4395d-f72c-4db9-814b-719660fbdf87\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") "
Sep 10 23:19:48.887009 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.886977    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b4395d-f72c-4db9-814b-719660fbdf87-openshift-service-ca\") pod \"f2b4395d-f72c-4db9-814b-719660fbdf87\" (UID: \"f2b4395d-f72c-4db9-814b-719660fbdf87\") "
Sep 10 23:19:48.887382 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.887352    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b4395d-f72c-4db9-814b-719660fbdf87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2b4395d-f72c-4db9-814b-719660fbdf87" (UID: "f2b4395d-f72c-4db9-814b-719660fbdf87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:48.888962 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.888935    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2b4395d-f72c-4db9-814b-719660fbdf87" (UID: "f2b4395d-f72c-4db9-814b-719660fbdf87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:19:48.889355 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.889334    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b4395d-f72c-4db9-814b-719660fbdf87-kube-api-access-xvmcq" (OuterVolumeSpecName: "kube-api-access-xvmcq") pod "f2b4395d-f72c-4db9-814b-719660fbdf87" (UID: "f2b4395d-f72c-4db9-814b-719660fbdf87"). InnerVolumeSpecName "kube-api-access-xvmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:48.889422 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.889363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2b4395d-f72c-4db9-814b-719660fbdf87" (UID: "f2b4395d-f72c-4db9-814b-719660fbdf87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:19:48.987789 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.987757    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmcq\" (UniqueName: \"kubernetes.io/projected/f2b4395d-f72c-4db9-814b-719660fbdf87-kube-api-access-xvmcq\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:48.987789 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.987784    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:48.987789 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.987794    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b4395d-f72c-4db9-814b-719660fbdf87-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:48.988006 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:48.987802    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b4395d-f72c-4db9-814b-719660fbdf87-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:19:49.288189 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.288156    2564 generic.go:356] "Generic (PLEG): container finished" podID="f2b4395d-f72c-4db9-814b-719660fbdf87" containerID="d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df" exitCode=0
Sep 10 23:19:49.288347 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.288197    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"
Sep 10 23:19:49.288347 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.288237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" event={"ID":"f2b4395d-f72c-4db9-814b-719660fbdf87","Type":"ContainerDied","Data":"d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df"}
Sep 10 23:19:49.288347 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.288293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6" event={"ID":"f2b4395d-f72c-4db9-814b-719660fbdf87","Type":"ContainerDied","Data":"35fd583d54c4376e6161c514c63f4bbaed4dee6cef95889ea04405b312a14b6c"}
Sep 10 23:19:49.288347 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.288321    2564 scope.go:117] "RemoveContainer" containerID="d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df"
Sep 10 23:19:49.297464 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.297447    2564 scope.go:117] "RemoveContainer" containerID="d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df"
Sep 10 23:19:49.297741 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:19:49.297722    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df\": container with ID starting with d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df not found: ID does not exist" containerID="d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df"
Sep 10 23:19:49.297802 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.297750    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df"} err="failed to get container status \"d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df\": rpc error: code = NotFound desc = could not find container \"d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df\": container with ID starting with d83d448d87f059a99ae64c5e46b3bee3eac880e769bfa7ef0ef338e2d3b323df not found: ID does not exist"
Sep 10 23:19:49.309943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.309921    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"]
Sep 10 23:19:49.314762 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:49.314741    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b3d9be-6888c7b4b7-m5xt6"]
Sep 10 23:19:50.198718 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:50.198676    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b4395d-f72c-4db9-814b-719660fbdf87" path="/var/lib/kubelet/pods/f2b4395d-f72c-4db9-814b-719660fbdf87/volumes"
Sep 10 23:19:52.255080 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:52.255046    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:19:52.255440 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:19:52.255109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:20:06.532578 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.532548    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"]
Sep 10 23:20:06.532984 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.532971    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b4395d-f72c-4db9-814b-719660fbdf87" containerName="maas-api"
Sep 10 23:20:06.533039 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.532986    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b4395d-f72c-4db9-814b-719660fbdf87" containerName="maas-api"
Sep 10 23:20:06.533085 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.533075    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b4395d-f72c-4db9-814b-719660fbdf87" containerName="maas-api"
Sep 10 23:20:06.536833 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.536812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.539594 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.539574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-7b202f-serving-cert\""
Sep 10 23:20:06.550845 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.550674    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"]
Sep 10 23:20:06.636257 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.636233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.636381 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.636279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.636381 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.636308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb995bd-999b-4213-ba03-10e2a0da3198-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.636462 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.636400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574bm\" (UniqueName: \"kubernetes.io/projected/adb995bd-999b-4213-ba03-10e2a0da3198-kube-api-access-574bm\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.737649 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.737616    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.737805 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.737662    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb995bd-999b-4213-ba03-10e2a0da3198-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.737805 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.737695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574bm\" (UniqueName: \"kubernetes.io/projected/adb995bd-999b-4213-ba03-10e2a0da3198-kube-api-access-574bm\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.737805 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.737767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.737909 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:20:06.737878    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-serving-cert: secret "maas-api-e2e-shared-b-w2-7b202f-serving-cert" not found
Sep 10 23:20:06.737956 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:20:06.737946    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls podName:adb995bd-999b-4213-ba03-10e2a0da3198 nodeName:}" failed. No retries permitted until 2026-09-10 23:20:07.237926996 +0000 UTC m=+1111.523948175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls") pod "maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" (UID: "adb995bd-999b-4213-ba03-10e2a0da3198") : secret "maas-api-e2e-shared-b-w2-7b202f-serving-cert" not found
Sep 10 23:20:06.738352 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.738333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb995bd-999b-4213-ba03-10e2a0da3198-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.740349 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.740329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:06.747673 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:06.747652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-574bm\" (UniqueName: \"kubernetes.io/projected/adb995bd-999b-4213-ba03-10e2a0da3198-kube-api-access-574bm\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:07.188807 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.188776    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"]
Sep 10 23:20:07.192190 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.192171    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.195087 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.195054    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7b202f-dockercfg-rlw8t\""
Sep 10 23:20:07.195208 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.195142    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7b202f\""
Sep 10 23:20:07.202127 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.202088    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"]
Sep 10 23:20:07.242008 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.241986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:07.242137 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.242023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ad6548-b3ac-4fec-bdca-8055f06378c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.242137 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.242040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztm8s\" (UniqueName: \"kubernetes.io/projected/72ad6548-b3ac-4fec-bdca-8055f06378c0-kube-api-access-ztm8s\") pod \"payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.244258 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.244238    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:07.343439 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.343402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ad6548-b3ac-4fec-bdca-8055f06378c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.343439 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.343443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztm8s\" (UniqueName: \"kubernetes.io/projected/72ad6548-b3ac-4fec-bdca-8055f06378c0-kube-api-access-ztm8s\") pod \"payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.344154 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.344136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ad6548-b3ac-4fec-bdca-8055f06378c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.353011 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.352989    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztm8s\" (UniqueName: \"kubernetes.io/projected/72ad6548-b3ac-4fec-bdca-8055f06378c0-kube-api-access-ztm8s\") pod \"payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.447225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.447167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:07.506070 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.506036    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:07.718646 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.718618    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"]
Sep 10 23:20:07.720425 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:20:07.720402    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb995bd_999b_4213_ba03_10e2a0da3198.slice/crio-ecddc2eaf3d13627a32c928ef2683b96ff2591936ddac97c34e1624a1adae77c WatchSource:0}: Error finding container ecddc2eaf3d13627a32c928ef2683b96ff2591936ddac97c34e1624a1adae77c: Status 404 returned error can't find the container with id ecddc2eaf3d13627a32c928ef2683b96ff2591936ddac97c34e1624a1adae77c
Sep 10 23:20:07.784158 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:07.784132    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"]
Sep 10 23:20:07.786817 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:20:07.786788    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ad6548_b3ac_4fec_bdca_8055f06378c0.slice/crio-e290248ddb9d0c6d8c9889bc4880001de495518d518fabdc3d54cf7dd87bb75a WatchSource:0}: Error finding container e290248ddb9d0c6d8c9889bc4880001de495518d518fabdc3d54cf7dd87bb75a: Status 404 returned error can't find the container with id e290248ddb9d0c6d8c9889bc4880001de495518d518fabdc3d54cf7dd87bb75a
Sep 10 23:20:08.387421 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.387324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" event={"ID":"adb995bd-999b-4213-ba03-10e2a0da3198","Type":"ContainerStarted","Data":"5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3"}
Sep 10 23:20:08.387421 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.387390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" event={"ID":"adb995bd-999b-4213-ba03-10e2a0da3198","Type":"ContainerStarted","Data":"ecddc2eaf3d13627a32c928ef2683b96ff2591936ddac97c34e1624a1adae77c"}
Sep 10 23:20:08.387632 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.387525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:08.389062 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.389035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh" event={"ID":"72ad6548-b3ac-4fec-bdca-8055f06378c0","Type":"ContainerStarted","Data":"1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c"}
Sep 10 23:20:08.389176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.389074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh" event={"ID":"72ad6548-b3ac-4fec-bdca-8055f06378c0","Type":"ContainerStarted","Data":"e290248ddb9d0c6d8c9889bc4880001de495518d518fabdc3d54cf7dd87bb75a"}
Sep 10 23:20:08.389176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.389124    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:08.403254 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.403217    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" podStartSLOduration=2.023515292 podStartE2EDuration="2.403206771s" totalImagesPullingTime="379.691479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:08.402462014 +0000 UTC m=+1112.688483215" watchObservedRunningTime="2026-09-10 23:20:08.403206771 +0000 UTC m=+1112.689227972"
Sep 10 23:20:08.418315 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:08.418275    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh" podStartSLOduration=1.418263177 podStartE2EDuration="1.418263177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:08.41639567 +0000 UTC m=+1112.702416871" watchObservedRunningTime="2026-09-10 23:20:08.418263177 +0000 UTC m=+1112.704284379"
Sep 10 23:20:14.399761 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:14.399700    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:20:19.396392 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:19.396365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:20:45.585671 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.585637    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"]
Sep 10 23:20:45.590928 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.590907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.593863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.593490    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-76145153-dockercfg-45626\""
Sep 10 23:20:45.593863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.593529    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-76145153\""
Sep 10 23:20:45.597776 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.597755    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"]
Sep 10 23:20:45.781515 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.781481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3529f801-4977-4cd0-88df-f4823f912ac2-plugins-config-volume\") pod \"payload-processing-e2e-mt-76145153-78dfb9457-hlkbl\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") " pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.781689 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.781534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzqt\" (UniqueName: \"kubernetes.io/projected/3529f801-4977-4cd0-88df-f4823f912ac2-kube-api-access-cmzqt\") pod \"payload-processing-e2e-mt-76145153-78dfb9457-hlkbl\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") " pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.882191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.882113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3529f801-4977-4cd0-88df-f4823f912ac2-plugins-config-volume\") pod \"payload-processing-e2e-mt-76145153-78dfb9457-hlkbl\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") " pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.882191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.882165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzqt\" (UniqueName: \"kubernetes.io/projected/3529f801-4977-4cd0-88df-f4823f912ac2-kube-api-access-cmzqt\") pod \"payload-processing-e2e-mt-76145153-78dfb9457-hlkbl\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") " pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.882828 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.882810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3529f801-4977-4cd0-88df-f4823f912ac2-plugins-config-volume\") pod \"payload-processing-e2e-mt-76145153-78dfb9457-hlkbl\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") " pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.889660 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.889638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzqt\" (UniqueName: \"kubernetes.io/projected/3529f801-4977-4cd0-88df-f4823f912ac2-kube-api-access-cmzqt\") pod \"payload-processing-e2e-mt-76145153-78dfb9457-hlkbl\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") " pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:45.904453 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:45.904428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:46.175841 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:46.175816    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"]
Sep 10 23:20:46.178044 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:20:46.178016    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3529f801_4977_4cd0_88df_f4823f912ac2.slice/crio-17be77bcefeff24339dcce54a8df9baa02216c4e8c96876e56d6b446699be59b WatchSource:0}: Error finding container 17be77bcefeff24339dcce54a8df9baa02216c4e8c96876e56d6b446699be59b: Status 404 returned error can't find the container with id 17be77bcefeff24339dcce54a8df9baa02216c4e8c96876e56d6b446699be59b
Sep 10 23:20:46.586154 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:46.586070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl" event={"ID":"3529f801-4977-4cd0-88df-f4823f912ac2","Type":"ContainerStarted","Data":"b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625"}
Sep 10 23:20:46.586154 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:46.586131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl" event={"ID":"3529f801-4977-4cd0-88df-f4823f912ac2","Type":"ContainerStarted","Data":"17be77bcefeff24339dcce54a8df9baa02216c4e8c96876e56d6b446699be59b"}
Sep 10 23:20:46.586547 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:46.586172    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:20:46.627400 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:46.627355    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl" podStartSLOduration=1.627339479 podStartE2EDuration="1.627339479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:46.626464792 +0000 UTC m=+1150.912485994" watchObservedRunningTime="2026-09-10 23:20:46.627339479 +0000 UTC m=+1150.913360681"
Sep 10 23:20:54.119828 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.119788    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"]
Sep 10 23:20:54.120269 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.120048    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4" podUID="20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" containerName="maas-api" containerID="cri-o://165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763" gracePeriod=30
Sep 10 23:20:54.381533 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.381470    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:20:54.450233 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.450208    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-maas-api-tls\") pod \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") "
Sep 10 23:20:54.450388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.450240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-metrics-certs\") pod \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") "
Sep 10 23:20:54.450388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.450269    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq4wg\" (UniqueName: \"kubernetes.io/projected/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-kube-api-access-dq4wg\") pod \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") "
Sep 10 23:20:54.450388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.450376    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-openshift-service-ca\") pod \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\" (UID: \"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c\") "
Sep 10 23:20:54.450758 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.450699    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" (UID: "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:20:54.452350 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.452323    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" (UID: "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:20:54.452350 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.452341    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" (UID: "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:20:54.452472 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.452407    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-kube-api-access-dq4wg" (OuterVolumeSpecName: "kube-api-access-dq4wg") pod "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" (UID: "20922ff7-5cda-46b6-bd9b-eb7fc9e9873c"). InnerVolumeSpecName "kube-api-access-dq4wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:54.551508 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.551480    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.551508 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.551504    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.551508 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.551513    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq4wg\" (UniqueName: \"kubernetes.io/projected/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-kube-api-access-dq4wg\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.551732 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.551523    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.629819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.629792    2564 generic.go:356] "Generic (PLEG): container finished" podID="20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" containerID="165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763" exitCode=0
Sep 10 23:20:54.629963 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.629832    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"
Sep 10 23:20:54.629963 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.629871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4" event={"ID":"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c","Type":"ContainerDied","Data":"165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763"}
Sep 10 23:20:54.629963 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.629924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4" event={"ID":"20922ff7-5cda-46b6-bd9b-eb7fc9e9873c","Type":"ContainerDied","Data":"cce52eccbb78ac117251d3bc81bd4fc729329dc1d9b7b590cdac8c6fea14cc39"}
Sep 10 23:20:54.629963 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.629947    2564 scope.go:117] "RemoveContainer" containerID="165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763"
Sep 10 23:20:54.639527 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.639509    2564 scope.go:117] "RemoveContainer" containerID="165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763"
Sep 10 23:20:54.639858 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:20:54.639839    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763\": container with ID starting with 165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763 not found: ID does not exist" containerID="165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763"
Sep 10 23:20:54.639917 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.639867    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763"} err="failed to get container status \"165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763\": rpc error: code = NotFound desc = could not find container \"165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763\": container with ID starting with 165f8d920e177c96b766588f26fa8c8d48c3052d22e9eb6e2de33f24df033763 not found: ID does not exist"
Sep 10 23:20:54.650682 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.650660    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"]
Sep 10 23:20:54.652560 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:54.652540    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-679c94-bfcb7c49c-sjfx4"]
Sep 10 23:20:56.200027 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:56.199996    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" path="/var/lib/kubelet/pods/20922ff7-5cda-46b6-bd9b-eb7fc9e9873c/volumes"
Sep 10 23:20:57.593795 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:20:57.593765    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:21:32.868593 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:32.868559    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"]
Sep 10 23:21:32.869129 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:32.868884    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" podUID="84ca9abb-323e-4bf4-a63b-1ee5232a0364" containerName="maas-api" containerID="cri-o://405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045" gracePeriod=30
Sep 10 23:21:33.122731 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.122635    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:21:33.278526 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.278493    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-metrics-certs\") pod \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") "
Sep 10 23:21:33.278526 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.278539    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h79hk\" (UniqueName: \"kubernetes.io/projected/84ca9abb-323e-4bf4-a63b-1ee5232a0364-kube-api-access-h79hk\") pod \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") "
Sep 10 23:21:33.278766 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.278592    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls\") pod \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") "
Sep 10 23:21:33.278766 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.278631    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ca9abb-323e-4bf4-a63b-1ee5232a0364-openshift-service-ca\") pod \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\" (UID: \"84ca9abb-323e-4bf4-a63b-1ee5232a0364\") "
Sep 10 23:21:33.279081 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.279048    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ca9abb-323e-4bf4-a63b-1ee5232a0364-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84ca9abb-323e-4bf4-a63b-1ee5232a0364" (UID: "84ca9abb-323e-4bf4-a63b-1ee5232a0364"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:33.280938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.280909    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84ca9abb-323e-4bf4-a63b-1ee5232a0364" (UID: "84ca9abb-323e-4bf4-a63b-1ee5232a0364"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:33.281229 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.281211    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ca9abb-323e-4bf4-a63b-1ee5232a0364-kube-api-access-h79hk" (OuterVolumeSpecName: "kube-api-access-h79hk") pod "84ca9abb-323e-4bf4-a63b-1ee5232a0364" (UID: "84ca9abb-323e-4bf4-a63b-1ee5232a0364"). InnerVolumeSpecName "kube-api-access-h79hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:33.281281 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.281222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84ca9abb-323e-4bf4-a63b-1ee5232a0364" (UID: "84ca9abb-323e-4bf4-a63b-1ee5232a0364"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:33.379442 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.379379    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.379442 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.379402    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ca9abb-323e-4bf4-a63b-1ee5232a0364-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.379442 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.379413    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84ca9abb-323e-4bf4-a63b-1ee5232a0364-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.379442 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.379421    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79hk\" (UniqueName: \"kubernetes.io/projected/84ca9abb-323e-4bf4-a63b-1ee5232a0364-kube-api-access-h79hk\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.837757 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.837723    2564 generic.go:356] "Generic (PLEG): container finished" podID="84ca9abb-323e-4bf4-a63b-1ee5232a0364" containerID="405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045" exitCode=0
Sep 10 23:21:33.837915 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.837770    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"
Sep 10 23:21:33.837915 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.837811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" event={"ID":"84ca9abb-323e-4bf4-a63b-1ee5232a0364","Type":"ContainerDied","Data":"405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045"}
Sep 10 23:21:33.837915 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.837867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw" event={"ID":"84ca9abb-323e-4bf4-a63b-1ee5232a0364","Type":"ContainerDied","Data":"c113acf68ff8ca338625c1bdfa1c5d68e7846b8371352a4961c0807283feeb9f"}
Sep 10 23:21:33.837915 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.837889    2564 scope.go:117] "RemoveContainer" containerID="405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045"
Sep 10 23:21:33.846895 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.846869    2564 scope.go:117] "RemoveContainer" containerID="405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045"
Sep 10 23:21:33.847171 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:21:33.847154    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045\": container with ID starting with 405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045 not found: ID does not exist" containerID="405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045"
Sep 10 23:21:33.847222 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.847180    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045"} err="failed to get container status \"405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045\": rpc error: code = NotFound desc = could not find container \"405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045\": container with ID starting with 405e7ef76802f2f1006e8f0846f27401280bba9f7fa264d9895e79f2a6589045 not found: ID does not exist"
Sep 10 23:21:33.859761 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.859729    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"]
Sep 10 23:21:33.863311 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:33.863292    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce17e-7f74bbdc9f-2jplw"]
Sep 10 23:21:34.199506 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:34.199475    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ca9abb-323e-4bf4-a63b-1ee5232a0364" path="/var/lib/kubelet/pods/84ca9abb-323e-4bf4-a63b-1ee5232a0364/volumes"
Sep 10 23:21:36.182507 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:36.182484    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:21:36.184919 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:36.184898    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:21:47.757936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:47.757897    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"]
Sep 10 23:21:47.758412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:47.758133    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq" podUID="d455f2e5-9f51-4947-af25-b4ced04337fe" containerName="maas-api" containerID="cri-o://482bb68f232ce442daedddc1f977a6aaf340d8f47e44a7d5553ef020101ca10d" gracePeriod=30
Sep 10 23:21:47.811183 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:47.811153    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"]
Sep 10 23:21:47.811414 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:47.811389    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g" podUID="b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" containerName="payload-pre-processing" containerID="cri-o://3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264" gracePeriod=30
Sep 10 23:21:47.911904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:47.911875    2564 generic.go:356] "Generic (PLEG): container finished" podID="d455f2e5-9f51-4947-af25-b4ced04337fe" containerID="482bb68f232ce442daedddc1f977a6aaf340d8f47e44a7d5553ef020101ca10d" exitCode=0
Sep 10 23:21:47.912124 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:47.911923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq" event={"ID":"d455f2e5-9f51-4947-af25-b4ced04337fe","Type":"ContainerDied","Data":"482bb68f232ce442daedddc1f977a6aaf340d8f47e44a7d5553ef020101ca10d"}
Sep 10 23:21:48.060010 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.059979    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:21:48.091669 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.091649    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:21:48.113067 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113036    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-plugins-config-volume\") pod \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") "
Sep 10 23:21:48.113216 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113125    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d455f2e5-9f51-4947-af25-b4ced04337fe-openshift-service-ca\") pod \"d455f2e5-9f51-4947-af25-b4ced04337fe\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") "
Sep 10 23:21:48.113216 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113194    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-metrics-certs\") pod \"d455f2e5-9f51-4947-af25-b4ced04337fe\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") "
Sep 10 23:21:48.113340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqjb7\" (UniqueName: \"kubernetes.io/projected/d455f2e5-9f51-4947-af25-b4ced04337fe-kube-api-access-wqjb7\") pod \"d455f2e5-9f51-4947-af25-b4ced04337fe\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") "
Sep 10 23:21:48.113340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113288    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") pod \"d455f2e5-9f51-4947-af25-b4ced04337fe\" (UID: \"d455f2e5-9f51-4947-af25-b4ced04337fe\") "
Sep 10 23:21:48.113340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113310    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpchk\" (UniqueName: \"kubernetes.io/projected/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-kube-api-access-tpchk\") pod \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\" (UID: \"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d\") "
Sep 10 23:21:48.113496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113453    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" (UID: "b1c0b48e-fcfb-4164-b4f6-f29a04b7348d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:48.113549 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113526    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d455f2e5-9f51-4947-af25-b4ced04337fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d455f2e5-9f51-4947-af25-b4ced04337fe" (UID: "d455f2e5-9f51-4947-af25-b4ced04337fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:48.113755 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113672    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d455f2e5-9f51-4947-af25-b4ced04337fe-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.113755 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.113695    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.115776 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.115737    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d455f2e5-9f51-4947-af25-b4ced04337fe-kube-api-access-wqjb7" (OuterVolumeSpecName: "kube-api-access-wqjb7") pod "d455f2e5-9f51-4947-af25-b4ced04337fe" (UID: "d455f2e5-9f51-4947-af25-b4ced04337fe"). InnerVolumeSpecName "kube-api-access-wqjb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:48.115877 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.115797    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d455f2e5-9f51-4947-af25-b4ced04337fe" (UID: "d455f2e5-9f51-4947-af25-b4ced04337fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:48.115982 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.115959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-kube-api-access-tpchk" (OuterVolumeSpecName: "kube-api-access-tpchk") pod "b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" (UID: "b1c0b48e-fcfb-4164-b4f6-f29a04b7348d"). InnerVolumeSpecName "kube-api-access-tpchk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:48.116022 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.115981    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d455f2e5-9f51-4947-af25-b4ced04337fe" (UID: "d455f2e5-9f51-4947-af25-b4ced04337fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:48.214350 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.214317    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.214350 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.214351    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqjb7\" (UniqueName: \"kubernetes.io/projected/d455f2e5-9f51-4947-af25-b4ced04337fe-kube-api-access-wqjb7\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.214350 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.214361    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d455f2e5-9f51-4947-af25-b4ced04337fe-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.214569 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.214371    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpchk\" (UniqueName: \"kubernetes.io/projected/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d-kube-api-access-tpchk\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.917561 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.917530    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" containerID="3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264" exitCode=0
Sep 10 23:21:48.917970 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.917569    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"
Sep 10 23:21:48.917970 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.917618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g" event={"ID":"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d","Type":"ContainerDied","Data":"3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264"}
Sep 10 23:21:48.917970 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.917677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g" event={"ID":"b1c0b48e-fcfb-4164-b4f6-f29a04b7348d","Type":"ContainerDied","Data":"b091b2aa5a44b5456c8c05b3150fbde7d9708b2c4de98bac3ddcbda5a60f7678"}
Sep 10 23:21:48.917970 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.917731    2564 scope.go:117] "RemoveContainer" containerID="3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264"
Sep 10 23:21:48.919248 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.919226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq" event={"ID":"d455f2e5-9f51-4947-af25-b4ced04337fe","Type":"ContainerDied","Data":"4799f154087a5921dc5f144eb41b809034e2820278ab3da23b13eb74372cfc4a"}
Sep 10 23:21:48.919333 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.919264    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"
Sep 10 23:21:48.926798 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.926782    2564 scope.go:117] "RemoveContainer" containerID="3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264"
Sep 10 23:21:48.927051 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:21:48.927025    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264\": container with ID starting with 3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264 not found: ID does not exist" containerID="3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264"
Sep 10 23:21:48.927142 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.927049    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264"} err="failed to get container status \"3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264\": rpc error: code = NotFound desc = could not find container \"3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264\": container with ID starting with 3d1f3413e2b00fa8767ba7557e99615f45bfa7fafb2c2dfb1d27baf496426264 not found: ID does not exist"
Sep 10 23:21:48.927142 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.927065    2564 scope.go:117] "RemoveContainer" containerID="482bb68f232ce442daedddc1f977a6aaf340d8f47e44a7d5553ef020101ca10d"
Sep 10 23:21:48.935861 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.935842    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"]
Sep 10 23:21:48.941727 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.941692    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-568856-578dd574d6-277sq"]
Sep 10 23:21:48.950780 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.950761    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"]
Sep 10 23:21:48.954293 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:48.954273    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-568856-678b76cc8-zbp9g"]
Sep 10 23:21:50.198834 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:50.198799    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" path="/var/lib/kubelet/pods/b1c0b48e-fcfb-4164-b4f6-f29a04b7348d/volumes"
Sep 10 23:21:50.199178 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:21:50.199164    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d455f2e5-9f51-4947-af25-b4ced04337fe" path="/var/lib/kubelet/pods/d455f2e5-9f51-4947-af25-b4ced04337fe/volumes"
Sep 10 23:23:36.055175 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055144    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"]
Sep 10 23:23:36.055885 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055859    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ca9abb-323e-4bf4-a63b-1ee5232a0364" containerName="maas-api"
Sep 10 23:23:36.055885 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055884    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ca9abb-323e-4bf4-a63b-1ee5232a0364" containerName="maas-api"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055900    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" containerName="payload-pre-processing"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055905    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" containerName="payload-pre-processing"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055928    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" containerName="maas-api"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055936    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" containerName="maas-api"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055952    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d455f2e5-9f51-4947-af25-b4ced04337fe" containerName="maas-api"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.055960    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d455f2e5-9f51-4947-af25-b4ced04337fe" containerName="maas-api"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.056062    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d455f2e5-9f51-4947-af25-b4ced04337fe" containerName="maas-api"
Sep 10 23:23:36.056072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.056075    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="20922ff7-5cda-46b6-bd9b-eb7fc9e9873c" containerName="maas-api"
Sep 10 23:23:36.056397 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.056084    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c0b48e-fcfb-4164-b4f6-f29a04b7348d" containerName="payload-pre-processing"
Sep 10 23:23:36.056397 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.056095    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ca9abb-323e-4bf4-a63b-1ee5232a0364" containerName="maas-api"
Sep 10 23:23:36.059075 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.059057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.061213 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.061192    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"kube-root-ca.crt\""
Sep 10 23:23:36.061972 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.061952    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"unconfigured-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:23:36.061972 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.061962    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"openshift-service-ca.crt\""
Sep 10 23:23:36.062142 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.062047    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"default-dockercfg-2lgbw\""
Sep 10 23:23:36.068448 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.068429    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"]
Sep 10 23:23:36.224140 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.224104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-dshm\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.224403 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.224370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552743e7-bc73-4d0b-b705-575c3e521a6e-tls-certs\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.224531 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.224432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5c9\" (UniqueName: \"kubernetes.io/projected/552743e7-bc73-4d0b-b705-575c3e521a6e-kube-api-access-ml5c9\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.224531 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.224464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-home\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.224531 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.224495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-model-cache\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.224531 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.224522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-tmp-dir\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.325774 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.325664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-tmp-dir\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.325947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.325823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-dshm\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.325947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.325861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552743e7-bc73-4d0b-b705-575c3e521a6e-tls-certs\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.325947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.325912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5c9\" (UniqueName: \"kubernetes.io/projected/552743e7-bc73-4d0b-b705-575c3e521a6e-kube-api-access-ml5c9\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.325947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.325944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-home\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.326159 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.325978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-model-cache\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.326159 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.326084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-tmp-dir\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.326311 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.326290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-model-cache\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.326372 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.326336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-home\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.327961 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.327938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-dshm\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.328138 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.328121    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"unconfigured-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:23:36.333149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.333129    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"kube-root-ca.crt\""
Sep 10 23:23:36.339144 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.339121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552743e7-bc73-4d0b-b705-575c3e521a6e-tls-certs\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.345839 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.345818    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"openshift-service-ca.crt\""
Sep 10 23:23:36.355741 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.355700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5c9\" (UniqueName: \"kubernetes.io/projected/552743e7-bc73-4d0b-b705-575c3e521a6e-kube-api-access-ml5c9\") pod \"unconfigured-w4-7025b9-kserve-855966678c-xxh2d\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") " pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.371681 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.371664    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"default-dockercfg-2lgbw\""
Sep 10 23:23:36.379919 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.379902    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:36.653253 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:36.653228    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"]
Sep 10 23:23:36.655119 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:23:36.655091    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod552743e7_bc73_4d0b_b705_575c3e521a6e.slice/crio-a2b6b6d39c5d200bec5cf5921e50a3ee342b703a20088cf717b1ab88038e6ccd WatchSource:0}: Error finding container a2b6b6d39c5d200bec5cf5921e50a3ee342b703a20088cf717b1ab88038e6ccd: Status 404 returned error can't find the container with id a2b6b6d39c5d200bec5cf5921e50a3ee342b703a20088cf717b1ab88038e6ccd
Sep 10 23:23:37.476346 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:37.476311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d" event={"ID":"552743e7-bc73-4d0b-b705-575c3e521a6e","Type":"ContainerStarted","Data":"6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e"}
Sep 10 23:23:37.476748 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:37.476361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d" event={"ID":"552743e7-bc73-4d0b-b705-575c3e521a6e","Type":"ContainerStarted","Data":"a2b6b6d39c5d200bec5cf5921e50a3ee342b703a20088cf717b1ab88038e6ccd"}
Sep 10 23:23:37.491996 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:37.491953    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d" podStartSLOduration=1.491938413 podStartE2EDuration="1.491938413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:23:37.490991869 +0000 UTC m=+1321.777013070" watchObservedRunningTime="2026-09-10 23:23:37.491938413 +0000 UTC m=+1321.777959672"
Sep 10 23:23:46.380109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:46.380079    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:46.380598 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:46.380133    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:46.392759 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:46.392732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:46.532751 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:46.532724    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:23:51.752584 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:51.752552    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"]
Sep 10 23:23:51.753219 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:51.753185    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl" podUID="3529f801-4977-4cd0-88df-f4823f912ac2" containerName="payload-processing" containerID="cri-o://b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625" gracePeriod=30
Sep 10 23:23:52.023268 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.023244    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:23:52.059203 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.059154    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmzqt\" (UniqueName: \"kubernetes.io/projected/3529f801-4977-4cd0-88df-f4823f912ac2-kube-api-access-cmzqt\") pod \"3529f801-4977-4cd0-88df-f4823f912ac2\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") "
Sep 10 23:23:52.059386 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.059213    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3529f801-4977-4cd0-88df-f4823f912ac2-plugins-config-volume\") pod \"3529f801-4977-4cd0-88df-f4823f912ac2\" (UID: \"3529f801-4977-4cd0-88df-f4823f912ac2\") "
Sep 10 23:23:52.059644 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.059615    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3529f801-4977-4cd0-88df-f4823f912ac2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3529f801-4977-4cd0-88df-f4823f912ac2" (UID: "3529f801-4977-4cd0-88df-f4823f912ac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:23:52.061319 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.061296    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3529f801-4977-4cd0-88df-f4823f912ac2-kube-api-access-cmzqt" (OuterVolumeSpecName: "kube-api-access-cmzqt") pod "3529f801-4977-4cd0-88df-f4823f912ac2" (UID: "3529f801-4977-4cd0-88df-f4823f912ac2"). InnerVolumeSpecName "kube-api-access-cmzqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:23:52.167545 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.167511    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3529f801-4977-4cd0-88df-f4823f912ac2-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.167545 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.167540    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzqt\" (UniqueName: \"kubernetes.io/projected/3529f801-4977-4cd0-88df-f4823f912ac2-kube-api-access-cmzqt\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.552019 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.551986    2564 generic.go:356] "Generic (PLEG): container finished" podID="3529f801-4977-4cd0-88df-f4823f912ac2" containerID="b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625" exitCode=0
Sep 10 23:23:52.552199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.552041    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl" event={"ID":"3529f801-4977-4cd0-88df-f4823f912ac2","Type":"ContainerDied","Data":"b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625"}
Sep 10 23:23:52.552199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.552053    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"
Sep 10 23:23:52.552199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.552102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl" event={"ID":"3529f801-4977-4cd0-88df-f4823f912ac2","Type":"ContainerDied","Data":"17be77bcefeff24339dcce54a8df9baa02216c4e8c96876e56d6b446699be59b"}
Sep 10 23:23:52.552199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.552130    2564 scope.go:117] "RemoveContainer" containerID="b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625"
Sep 10 23:23:52.560832 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.560814    2564 scope.go:117] "RemoveContainer" containerID="b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625"
Sep 10 23:23:52.561097 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:23:52.561075    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625\": container with ID starting with b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625 not found: ID does not exist" containerID="b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625"
Sep 10 23:23:52.561141 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.561105    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625"} err="failed to get container status \"b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625\": rpc error: code = NotFound desc = could not find container \"b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625\": container with ID starting with b220dd375165c05b48f1b28595703d486b15577ceca2a7620f0aec85cb6fe625 not found: ID does not exist"
Sep 10 23:23:52.567191 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.567170    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"]
Sep 10 23:23:52.573662 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:52.573640    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-76145153-78dfb9457-hlkbl"]
Sep 10 23:23:54.199141 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:23:54.199104    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3529f801-4977-4cd0-88df-f4823f912ac2" path="/var/lib/kubelet/pods/3529f801-4977-4cd0-88df-f4823f912ac2/volumes"
Sep 10 23:24:50.756645 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.756614    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"]
Sep 10 23:24:50.757089 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.757077    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3529f801-4977-4cd0-88df-f4823f912ac2" containerName="payload-processing"
Sep 10 23:24:50.757134 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.757092    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3529f801-4977-4cd0-88df-f4823f912ac2" containerName="payload-processing"
Sep 10 23:24:50.757180 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.757169    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3529f801-4977-4cd0-88df-f4823f912ac2" containerName="payload-processing"
Sep 10 23:24:50.761466 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.761449    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.764299 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.764277    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"kube-root-ca.crt\""
Sep 10 23:24:50.765225 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.765208    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"openshift-service-ca.crt\""
Sep 10 23:24:50.765340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.765225    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"default-dockercfg-dq6xc\""
Sep 10 23:24:50.765340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.765300    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"auth-test-model-83be66-kserve-self-signed-certs\""
Sep 10 23:24:50.771153 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.771130    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"]
Sep 10 23:24:50.846203 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.846175    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tmp-dir\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.846339 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.846218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-home\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.846339 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.846290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-dshm\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.846339 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.846320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tls-certs\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.846447 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.846344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-model-cache\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.846447 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.846368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7vf\" (UniqueName: \"kubernetes.io/projected/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-kube-api-access-jl7vf\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947163 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tls-certs\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947280 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-model-cache\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947280 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7vf\" (UniqueName: \"kubernetes.io/projected/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-kube-api-access-jl7vf\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947280 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tmp-dir\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947506 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-home\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947506 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-dshm\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947618 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-model-cache\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947675 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947630    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tmp-dir\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.947675 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.947662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-home\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.949447 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.949424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-dshm\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.949768 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.949748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tls-certs\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:50.956198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:50.956177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7vf\" (UniqueName: \"kubernetes.io/projected/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-kube-api-access-jl7vf\") pod \"auth-test-model-83be66-kserve-7b9976d947-k495t\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") " pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:51.071756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:51.071685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:24:51.341396 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:51.341375    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"]
Sep 10 23:24:51.343091 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:24:51.343064    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48dcdfb6_f950_48ee_ab5d_8f10d9478c10.slice/crio-847fd197b48af41f30a83e2ec715c7bef24e01f30a18728f9d90b45c689d41c2 WatchSource:0}: Error finding container 847fd197b48af41f30a83e2ec715c7bef24e01f30a18728f9d90b45c689d41c2: Status 404 returned error can't find the container with id 847fd197b48af41f30a83e2ec715c7bef24e01f30a18728f9d90b45c689d41c2
Sep 10 23:24:51.850802 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:51.850766    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t" event={"ID":"48dcdfb6-f950-48ee-ab5d-8f10d9478c10","Type":"ContainerStarted","Data":"5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85"}
Sep 10 23:24:51.850802 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:51.850809    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t" event={"ID":"48dcdfb6-f950-48ee-ab5d-8f10d9478c10","Type":"ContainerStarted","Data":"847fd197b48af41f30a83e2ec715c7bef24e01f30a18728f9d90b45c689d41c2"}
Sep 10 23:24:51.869336 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:24:51.869290    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t" podStartSLOduration=1.869274362 podStartE2EDuration="1.869274362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:24:51.867319179 +0000 UTC m=+1396.153340380" watchObservedRunningTime="2026-09-10 23:24:51.869274362 +0000 UTC m=+1396.155295562"
Sep 10 23:25:01.071902 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:01.071861    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:25:01.071902 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:01.071914    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:25:01.084616 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:01.084591    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:25:01.909483 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:01.909452    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:25:56.798924 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.798849    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"]
Sep 10 23:25:56.802548 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.802527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:56.804873 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.804853    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a4c77c-serving-cert\""
Sep 10 23:25:56.813672 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.813644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"]
Sep 10 23:25:56.920146 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.920103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:56.920319 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.920240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-metrics-certs\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:56.920319 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.920282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtt9x\" (UniqueName: \"kubernetes.io/projected/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-kube-api-access-mtt9x\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:56.920319 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:56.920304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-maas-api-tls\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.021536 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.021503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-metrics-certs\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.021536 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.021541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtt9x\" (UniqueName: \"kubernetes.io/projected/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-kube-api-access-mtt9x\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.021797 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.021561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-maas-api-tls\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.021797 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.021602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.022307 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.022287    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.024128 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.024097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-metrics-certs\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.024231 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.024107    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-maas-api-tls\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.029823 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.029805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtt9x\" (UniqueName: \"kubernetes.io/projected/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-kube-api-access-mtt9x\") pod \"maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.114628 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.114563    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:57.383439 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.383410    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"]
Sep 10 23:25:57.385317 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:25:57.385292    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a576bf_fe1a_423a_b922_1bb6f2477c0b.slice/crio-c49c5d89d712c780602cfe551c078091b20a4320f66385a7e5ccb8493d243b6a WatchSource:0}: Error finding container c49c5d89d712c780602cfe551c078091b20a4320f66385a7e5ccb8493d243b6a: Status 404 returned error can't find the container with id c49c5d89d712c780602cfe551c078091b20a4320f66385a7e5ccb8493d243b6a
Sep 10 23:25:57.386864 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.386844    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:25:57.438731 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.438692    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"]
Sep 10 23:25:57.443477 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.443453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.445853 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.445830    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a4c77c-dockercfg-75xkk\""
Sep 10 23:25:57.445973 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.445854    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a4c77c\""
Sep 10 23:25:57.450261 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.450232    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"]
Sep 10 23:25:57.527496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.527468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887cd0f5-ce74-421e-b641-17f013663d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.527611 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.527503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d725x\" (UniqueName: \"kubernetes.io/projected/887cd0f5-ce74-421e-b641-17f013663d34-kube-api-access-d725x\") pod \"payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.629156 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.629123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887cd0f5-ce74-421e-b641-17f013663d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.629296 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.629164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d725x\" (UniqueName: \"kubernetes.io/projected/887cd0f5-ce74-421e-b641-17f013663d34-kube-api-access-d725x\") pod \"payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.629810 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.629790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887cd0f5-ce74-421e-b641-17f013663d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.638154 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.638097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d725x\" (UniqueName: \"kubernetes.io/projected/887cd0f5-ce74-421e-b641-17f013663d34-kube-api-access-d725x\") pod \"payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:57.756994 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:57.756968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:58.040651 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.040617    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"]
Sep 10 23:25:58.042578 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:25:58.042550    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887cd0f5_ce74_421e_b641_17f013663d34.slice/crio-fdebd0cd0c6df2ccec36688273e93f09f849ac32a2ec92c351270cfbf5b73070 WatchSource:0}: Error finding container fdebd0cd0c6df2ccec36688273e93f09f849ac32a2ec92c351270cfbf5b73070: Status 404 returned error can't find the container with id fdebd0cd0c6df2ccec36688273e93f09f849ac32a2ec92c351270cfbf5b73070
Sep 10 23:25:58.190001 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.189968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn" event={"ID":"887cd0f5-ce74-421e-b641-17f013663d34","Type":"ContainerStarted","Data":"a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b"}
Sep 10 23:25:58.190176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.190018    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn" event={"ID":"887cd0f5-ce74-421e-b641-17f013663d34","Type":"ContainerStarted","Data":"fdebd0cd0c6df2ccec36688273e93f09f849ac32a2ec92c351270cfbf5b73070"}
Sep 10 23:25:58.190176 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.190065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:25:58.205904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.205433    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn" podStartSLOduration=1.205415451 podStartE2EDuration="1.205415451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:25:58.204688489 +0000 UTC m=+1462.490709686" watchObservedRunningTime="2026-09-10 23:25:58.205415451 +0000 UTC m=+1462.491436653"
Sep 10 23:25:58.208051 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.208022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:25:58.208190 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.208058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t" event={"ID":"b0a576bf-fe1a-423a-b922-1bb6f2477c0b","Type":"ContainerStarted","Data":"ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a"}
Sep 10 23:25:58.208190 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.208077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t" event={"ID":"b0a576bf-fe1a-423a-b922-1bb6f2477c0b","Type":"ContainerStarted","Data":"c49c5d89d712c780602cfe551c078091b20a4320f66385a7e5ccb8493d243b6a"}
Sep 10 23:25:58.219861 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:25:58.219822    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t" podStartSLOduration=1.839206584 podStartE2EDuration="2.21981035s" totalImagesPullingTime="380.603766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:25:58.219058288 +0000 UTC m=+1462.505079502" watchObservedRunningTime="2026-09-10 23:25:58.21981035 +0000 UTC m=+1462.505831550"
Sep 10 23:26:04.206947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:04.206921    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:26:09.198864 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:09.198830    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:26:25.999048 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:25.999014    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"]
Sep 10 23:26:26.002713 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.002681    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.005290 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.005270    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-72f076-serving-cert\""
Sep 10 23:26:26.011657 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.011632    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"]
Sep 10 23:26:26.083397 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.083370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58rx\" (UniqueName: \"kubernetes.io/projected/88cd0224-3942-4349-a9b6-7625dc50aed0-kube-api-access-m58rx\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.083539 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.083421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.083539 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.083483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-metrics-certs\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.083664 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.083595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88cd0224-3942-4349-a9b6-7625dc50aed0-openshift-service-ca\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.184533 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.184498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88cd0224-3942-4349-a9b6-7625dc50aed0-openshift-service-ca\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.184685 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.184553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58rx\" (UniqueName: \"kubernetes.io/projected/88cd0224-3942-4349-a9b6-7625dc50aed0-kube-api-access-m58rx\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.184685 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.184588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.184685 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.184631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-metrics-certs\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.184857 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:26:26.184753    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-serving-cert: secret "maas-api-e2e-api-b-72f076-serving-cert" not found
Sep 10 23:26:26.184857 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:26:26.184835    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls podName:88cd0224-3942-4349-a9b6-7625dc50aed0 nodeName:}" failed. No retries permitted until 2026-09-10 23:26:26.684814421 +0000 UTC m=+1490.970835600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls") pod "maas-api-e2e-api-b-72f076-fb965c995-69ctx" (UID: "88cd0224-3942-4349-a9b6-7625dc50aed0") : secret "maas-api-e2e-api-b-72f076-serving-cert" not found
Sep 10 23:26:26.185248 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.185231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88cd0224-3942-4349-a9b6-7625dc50aed0-openshift-service-ca\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.187151 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.187124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-metrics-certs\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.194612 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.194588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58rx\" (UniqueName: \"kubernetes.io/projected/88cd0224-3942-4349-a9b6-7625dc50aed0-kube-api-access-m58rx\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.635680 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.635650    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"]
Sep 10 23:26:26.639369 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.639353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.641612 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.641594    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-72f076-dockercfg-rrqtx\""
Sep 10 23:26:26.641713 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.641671    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-72f076\""
Sep 10 23:26:26.649611 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.649584    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"]
Sep 10 23:26:26.688458 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.688433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:26.688574 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.688470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98tg\" (UniqueName: \"kubernetes.io/projected/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-kube-api-access-p98tg\") pod \"payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.688574 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.688520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.688691 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:26:26.688591    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-serving-cert: secret "maas-api-e2e-api-b-72f076-serving-cert" not found
Sep 10 23:26:26.688691 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:26:26.688662    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls podName:88cd0224-3942-4349-a9b6-7625dc50aed0 nodeName:}" failed. No retries permitted until 2026-09-10 23:26:27.688642573 +0000 UTC m=+1491.974663766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls") pod "maas-api-e2e-api-b-72f076-fb965c995-69ctx" (UID: "88cd0224-3942-4349-a9b6-7625dc50aed0") : secret "maas-api-e2e-api-b-72f076-serving-cert" not found
Sep 10 23:26:26.789585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.789553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.789790 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.789691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98tg\" (UniqueName: \"kubernetes.io/projected/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-kube-api-access-p98tg\") pod \"payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.790344 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.790320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.797248 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.797225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98tg\" (UniqueName: \"kubernetes.io/projected/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-kube-api-access-p98tg\") pod \"payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:26.951871 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:26.951840    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:27.228572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.228542    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"]
Sep 10 23:26:27.229640 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:26:27.229615    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3ce8d3_a1f8_4f1e_ac55_623fb78313d2.slice/crio-4830adf5c846b395f2def610d4357c1d90b25c5a883727e7033df3348a477012 WatchSource:0}: Error finding container 4830adf5c846b395f2def610d4357c1d90b25c5a883727e7033df3348a477012: Status 404 returned error can't find the container with id 4830adf5c846b395f2def610d4357c1d90b25c5a883727e7033df3348a477012
Sep 10 23:26:27.337764 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.337723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px" event={"ID":"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2","Type":"ContainerStarted","Data":"639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096"}
Sep 10 23:26:27.337913 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.337793    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px" event={"ID":"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2","Type":"ContainerStarted","Data":"4830adf5c846b395f2def610d4357c1d90b25c5a883727e7033df3348a477012"}
Sep 10 23:26:27.337975 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.337917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:26:27.352879 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.352825    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px" podStartSLOduration=1.352808948 podStartE2EDuration="1.352808948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:26:27.352219252 +0000 UTC m=+1491.638240453" watchObservedRunningTime="2026-09-10 23:26:27.352808948 +0000 UTC m=+1491.638830150"
Sep 10 23:26:27.698024 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.697992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:27.700505 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.700474    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") pod \"maas-api-e2e-api-b-72f076-fb965c995-69ctx\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:27.816587 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:27.816558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:28.092310 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:28.092288    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"]
Sep 10 23:26:28.094341 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:26:28.094312    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cd0224_3942_4349_a9b6_7625dc50aed0.slice/crio-fbc46a823bdb1d0ae83cd58d91ab012987f02f0ef77d874d0f9e31d32d182302 WatchSource:0}: Error finding container fbc46a823bdb1d0ae83cd58d91ab012987f02f0ef77d874d0f9e31d32d182302: Status 404 returned error can't find the container with id fbc46a823bdb1d0ae83cd58d91ab012987f02f0ef77d874d0f9e31d32d182302
Sep 10 23:26:28.344938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:28.344834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx" event={"ID":"88cd0224-3942-4349-a9b6-7625dc50aed0","Type":"ContainerStarted","Data":"fbc46a823bdb1d0ae83cd58d91ab012987f02f0ef77d874d0f9e31d32d182302"}
Sep 10 23:26:29.352208 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:29.352172    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx" event={"ID":"88cd0224-3942-4349-a9b6-7625dc50aed0","Type":"ContainerStarted","Data":"8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff"}
Sep 10 23:26:29.352575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:29.352249    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:29.368669 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:29.368619    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx" podStartSLOduration=3.987673732 podStartE2EDuration="4.368602049s" totalImagesPullingTime="380.928317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:26:29.368545563 +0000 UTC m=+1493.654566763" watchObservedRunningTime="2026-09-10 23:26:29.368602049 +0000 UTC m=+1493.654623251"
Sep 10 23:26:35.361997 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:35.361972    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:26:36.216258 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:36.216233    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:26:36.228881 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:36.228851    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:26:38.347599 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:26:38.347567    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:27:08.065535 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.065449    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"]
Sep 10 23:27:08.068009 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.066286    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t" podUID="b0a576bf-fe1a-423a-b922-1bb6f2477c0b" containerName="maas-api" containerID="cri-o://ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a" gracePeriod=30
Sep 10 23:27:08.121980 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.121948    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"]
Sep 10 23:27:08.122195 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.122168    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn" podUID="887cd0f5-ce74-421e-b641-17f013663d34" containerName="payload-pre-processing" containerID="cri-o://a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b" gracePeriod=30
Sep 10 23:27:08.395674 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.395368    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:27:08.415205 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.415180    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:27:08.459632 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.459604    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d725x\" (UniqueName: \"kubernetes.io/projected/887cd0f5-ce74-421e-b641-17f013663d34-kube-api-access-d725x\") pod \"887cd0f5-ce74-421e-b641-17f013663d34\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") "
Sep 10 23:27:08.459784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.459668    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-openshift-service-ca\") pod \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") "
Sep 10 23:27:08.459784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.459746    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-metrics-certs\") pod \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") "
Sep 10 23:27:08.459784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.459765    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887cd0f5-ce74-421e-b641-17f013663d34-plugins-config-volume\") pod \"887cd0f5-ce74-421e-b641-17f013663d34\" (UID: \"887cd0f5-ce74-421e-b641-17f013663d34\") "
Sep 10 23:27:08.459892 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.459813    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtt9x\" (UniqueName: \"kubernetes.io/projected/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-kube-api-access-mtt9x\") pod \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") "
Sep 10 23:27:08.459892 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.459836    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-maas-api-tls\") pod \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\" (UID: \"b0a576bf-fe1a-423a-b922-1bb6f2477c0b\") "
Sep 10 23:27:08.460109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.460082    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0a576bf-fe1a-423a-b922-1bb6f2477c0b" (UID: "b0a576bf-fe1a-423a-b922-1bb6f2477c0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:08.460224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.460176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887cd0f5-ce74-421e-b641-17f013663d34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "887cd0f5-ce74-421e-b641-17f013663d34" (UID: "887cd0f5-ce74-421e-b641-17f013663d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:08.460224 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.460210    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.461968 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.461946    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0a576bf-fe1a-423a-b922-1bb6f2477c0b" (UID: "b0a576bf-fe1a-423a-b922-1bb6f2477c0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:08.461968 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.461960    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887cd0f5-ce74-421e-b641-17f013663d34-kube-api-access-d725x" (OuterVolumeSpecName: "kube-api-access-d725x") pod "887cd0f5-ce74-421e-b641-17f013663d34" (UID: "887cd0f5-ce74-421e-b641-17f013663d34"). InnerVolumeSpecName "kube-api-access-d725x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:08.462109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.461999    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0a576bf-fe1a-423a-b922-1bb6f2477c0b" (UID: "b0a576bf-fe1a-423a-b922-1bb6f2477c0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:08.462109 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.462008    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-kube-api-access-mtt9x" (OuterVolumeSpecName: "kube-api-access-mtt9x") pod "b0a576bf-fe1a-423a-b922-1bb6f2477c0b" (UID: "b0a576bf-fe1a-423a-b922-1bb6f2477c0b"). InnerVolumeSpecName "kube-api-access-mtt9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:08.561205 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.561182    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d725x\" (UniqueName: \"kubernetes.io/projected/887cd0f5-ce74-421e-b641-17f013663d34-kube-api-access-d725x\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.561205 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.561205    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.561349 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.561214    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887cd0f5-ce74-421e-b641-17f013663d34-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.561349 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.561223    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtt9x\" (UniqueName: \"kubernetes.io/projected/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-kube-api-access-mtt9x\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.561349 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.561231    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a576bf-fe1a-423a-b922-1bb6f2477c0b-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.566026 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.566000    2564 generic.go:356] "Generic (PLEG): container finished" podID="887cd0f5-ce74-421e-b641-17f013663d34" containerID="a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b" exitCode=0
Sep 10 23:27:08.566125 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.566041    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"
Sep 10 23:27:08.566125 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.566083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn" event={"ID":"887cd0f5-ce74-421e-b641-17f013663d34","Type":"ContainerDied","Data":"a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b"}
Sep 10 23:27:08.566243 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.566137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn" event={"ID":"887cd0f5-ce74-421e-b641-17f013663d34","Type":"ContainerDied","Data":"fdebd0cd0c6df2ccec36688273e93f09f849ac32a2ec92c351270cfbf5b73070"}
Sep 10 23:27:08.566243 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.566165    2564 scope.go:117] "RemoveContainer" containerID="a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b"
Sep 10 23:27:08.567728 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.567687    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0a576bf-fe1a-423a-b922-1bb6f2477c0b" containerID="ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a" exitCode=0
Sep 10 23:27:08.567851 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.567730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t" event={"ID":"b0a576bf-fe1a-423a-b922-1bb6f2477c0b","Type":"ContainerDied","Data":"ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a"}
Sep 10 23:27:08.567851 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.567759    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"
Sep 10 23:27:08.567851 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.567782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t" event={"ID":"b0a576bf-fe1a-423a-b922-1bb6f2477c0b","Type":"ContainerDied","Data":"c49c5d89d712c780602cfe551c078091b20a4320f66385a7e5ccb8493d243b6a"}
Sep 10 23:27:08.576737 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.576691    2564 scope.go:117] "RemoveContainer" containerID="a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b"
Sep 10 23:27:08.577033 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:27:08.577001    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b\": container with ID starting with a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b not found: ID does not exist" containerID="a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b"
Sep 10 23:27:08.577134 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.577042    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b"} err="failed to get container status \"a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b\": rpc error: code = NotFound desc = could not find container \"a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b\": container with ID starting with a8596b0c7130598b3af1989b37c2d84325da62054562d052b517894cf9867f9b not found: ID does not exist"
Sep 10 23:27:08.577134 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.577060    2564 scope.go:117] "RemoveContainer" containerID="ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a"
Sep 10 23:27:08.585273 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.585253    2564 scope.go:117] "RemoveContainer" containerID="ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a"
Sep 10 23:27:08.585526 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:27:08.585509    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a\": container with ID starting with ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a not found: ID does not exist" containerID="ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a"
Sep 10 23:27:08.585590 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.585532    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a"} err="failed to get container status \"ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a\": rpc error: code = NotFound desc = could not find container \"ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a\": container with ID starting with ae47dc425c1f00093e96a2055a8435d44efa7e88b71c341ec43d54d03e0d1a7a not found: ID does not exist"
Sep 10 23:27:08.591471 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.591448    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"]
Sep 10 23:27:08.593530 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.593509    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4c77c-7d646cb87d-fvjsn"]
Sep 10 23:27:08.600967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.600950    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"]
Sep 10 23:27:08.604237 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:08.604217    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4c77c-669c5d9bf5-jtm2t"]
Sep 10 23:27:10.199153 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:10.199108    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887cd0f5-ce74-421e-b641-17f013663d34" path="/var/lib/kubelet/pods/887cd0f5-ce74-421e-b641-17f013663d34/volumes"
Sep 10 23:27:10.199720 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:10.199685    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a576bf-fe1a-423a-b922-1bb6f2477c0b" path="/var/lib/kubelet/pods/b0a576bf-fe1a-423a-b922-1bb6f2477c0b/volumes"
Sep 10 23:27:30.649973 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:30.649937    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"]
Sep 10 23:27:30.650529 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:30.650235    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx" podUID="88cd0224-3942-4349-a9b6-7625dc50aed0" containerName="maas-api" containerID="cri-o://8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff" gracePeriod=30
Sep 10 23:27:30.704895 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:30.704860    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"]
Sep 10 23:27:30.705158 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:30.705107    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px" podUID="8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" containerName="payload-processing" containerID="cri-o://639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096" gracePeriod=30
Sep 10 23:27:30.982147 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:30.982125    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:27:31.011654 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.011630    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:27:31.055550 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.055526    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m58rx\" (UniqueName: \"kubernetes.io/projected/88cd0224-3942-4349-a9b6-7625dc50aed0-kube-api-access-m58rx\") pod \"88cd0224-3942-4349-a9b6-7625dc50aed0\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") "
Sep 10 23:27:31.055664 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.055564    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") pod \"88cd0224-3942-4349-a9b6-7625dc50aed0\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") "
Sep 10 23:27:31.055664 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.055648    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88cd0224-3942-4349-a9b6-7625dc50aed0-openshift-service-ca\") pod \"88cd0224-3942-4349-a9b6-7625dc50aed0\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") "
Sep 10 23:27:31.055762 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.055727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-metrics-certs\") pod \"88cd0224-3942-4349-a9b6-7625dc50aed0\" (UID: \"88cd0224-3942-4349-a9b6-7625dc50aed0\") "
Sep 10 23:27:31.056070 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.056041    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88cd0224-3942-4349-a9b6-7625dc50aed0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88cd0224-3942-4349-a9b6-7625dc50aed0" (UID: "88cd0224-3942-4349-a9b6-7625dc50aed0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:31.057542 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.057518    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cd0224-3942-4349-a9b6-7625dc50aed0-kube-api-access-m58rx" (OuterVolumeSpecName: "kube-api-access-m58rx") pod "88cd0224-3942-4349-a9b6-7625dc50aed0" (UID: "88cd0224-3942-4349-a9b6-7625dc50aed0"). InnerVolumeSpecName "kube-api-access-m58rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:31.057653 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.057616    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "88cd0224-3942-4349-a9b6-7625dc50aed0" (UID: "88cd0224-3942-4349-a9b6-7625dc50aed0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:31.057981 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.057959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88cd0224-3942-4349-a9b6-7625dc50aed0" (UID: "88cd0224-3942-4349-a9b6-7625dc50aed0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:31.157135 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.156962    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p98tg\" (UniqueName: \"kubernetes.io/projected/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-kube-api-access-p98tg\") pod \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") "
Sep 10 23:27:31.157135 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.157106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-plugins-config-volume\") pod \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\" (UID: \"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2\") "
Sep 10 23:27:31.157428 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.157408    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88cd0224-3942-4349-a9b6-7625dc50aed0-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.157489 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.157434    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.157489 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.157445    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58rx\" (UniqueName: \"kubernetes.io/projected/88cd0224-3942-4349-a9b6-7625dc50aed0-kube-api-access-m58rx\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.157489 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.157453    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88cd0224-3942-4349-a9b6-7625dc50aed0-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.157646 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.157528    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" (UID: "8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:31.159406 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.159376    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-kube-api-access-p98tg" (OuterVolumeSpecName: "kube-api-access-p98tg") pod "8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" (UID: "8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2"). InnerVolumeSpecName "kube-api-access-p98tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:31.258797 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.258736    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.258797 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.258759    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98tg\" (UniqueName: \"kubernetes.io/projected/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2-kube-api-access-p98tg\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.689923 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.689889    2564 generic.go:356] "Generic (PLEG): container finished" podID="8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" containerID="639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096" exitCode=0
Sep 10 23:27:31.690337 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.689937    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"
Sep 10 23:27:31.690337 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.689987    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px" event={"ID":"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2","Type":"ContainerDied","Data":"639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096"}
Sep 10 23:27:31.690337 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.690039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px" event={"ID":"8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2","Type":"ContainerDied","Data":"4830adf5c846b395f2def610d4357c1d90b25c5a883727e7033df3348a477012"}
Sep 10 23:27:31.690337 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.690083    2564 scope.go:117] "RemoveContainer" containerID="639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096"
Sep 10 23:27:31.691496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.691476    2564 generic.go:356] "Generic (PLEG): container finished" podID="88cd0224-3942-4349-a9b6-7625dc50aed0" containerID="8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff" exitCode=0
Sep 10 23:27:31.691572 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.691518    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"
Sep 10 23:27:31.691633 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.691564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx" event={"ID":"88cd0224-3942-4349-a9b6-7625dc50aed0","Type":"ContainerDied","Data":"8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff"}
Sep 10 23:27:31.691685 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.691629    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx" event={"ID":"88cd0224-3942-4349-a9b6-7625dc50aed0","Type":"ContainerDied","Data":"fbc46a823bdb1d0ae83cd58d91ab012987f02f0ef77d874d0f9e31d32d182302"}
Sep 10 23:27:31.704833 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.704819    2564 scope.go:117] "RemoveContainer" containerID="639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096"
Sep 10 23:27:31.705137 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:27:31.705116    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096\": container with ID starting with 639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096 not found: ID does not exist" containerID="639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096"
Sep 10 23:27:31.705188 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.705145    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096"} err="failed to get container status \"639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096\": rpc error: code = NotFound desc = could not find container \"639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096\": container with ID starting with 639b69c0fd42c949b185b3c35d202078529ff120d370710bf136c42636b2b096 not found: ID does not exist"
Sep 10 23:27:31.705188 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.705162    2564 scope.go:117] "RemoveContainer" containerID="8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff"
Sep 10 23:27:31.713874 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.713845    2564 scope.go:117] "RemoveContainer" containerID="8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff"
Sep 10 23:27:31.714140 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:27:31.714119    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff\": container with ID starting with 8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff not found: ID does not exist" containerID="8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff"
Sep 10 23:27:31.714199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.714145    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff"} err="failed to get container status \"8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff\": rpc error: code = NotFound desc = could not find container \"8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff\": container with ID starting with 8268fbd42a44fa362815b1e1351ad86e5594bef078bc213eb3f8c3d18d6ffbff not found: ID does not exist"
Sep 10 23:27:31.716131 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.716109    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"]
Sep 10 23:27:31.718425 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.718407    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-72f076-77d97bb65c-vd6px"]
Sep 10 23:27:31.726265 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.726244    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"]
Sep 10 23:27:31.731589 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:31.731568    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-72f076-fb965c995-69ctx"]
Sep 10 23:27:32.199650 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:32.199615    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88cd0224-3942-4349-a9b6-7625dc50aed0" path="/var/lib/kubelet/pods/88cd0224-3942-4349-a9b6-7625dc50aed0/volumes"
Sep 10 23:27:32.200054 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:32.200038    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" path="/var/lib/kubelet/pods/8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2/volumes"
Sep 10 23:27:48.219994 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.219962    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"]
Sep 10 23:27:48.220495 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.220222    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb" podUID="9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" containerName="payload-processing" containerID="cri-o://8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8" gracePeriod=30
Sep 10 23:27:48.225848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.225820    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"]
Sep 10 23:27:48.226047 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.226020    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6" podUID="f5d08644-a56b-4a00-83bd-d3f98bda474e" containerName="payload-pre-processing" containerID="cri-o://d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a" gracePeriod=30
Sep 10 23:27:48.511874 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.511843    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:27:48.514761 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.514740    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:27:48.708230 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.708197    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjcs4\" (UniqueName: \"kubernetes.io/projected/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-kube-api-access-xjcs4\") pod \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") "
Sep 10 23:27:48.708370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.708297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hldm\" (UniqueName: \"kubernetes.io/projected/f5d08644-a56b-4a00-83bd-d3f98bda474e-kube-api-access-6hldm\") pod \"f5d08644-a56b-4a00-83bd-d3f98bda474e\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") "
Sep 10 23:27:48.708370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.708341    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-plugins-config-volume\") pod \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\" (UID: \"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c\") "
Sep 10 23:27:48.708451 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.708385    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d08644-a56b-4a00-83bd-d3f98bda474e-plugins-config-volume\") pod \"f5d08644-a56b-4a00-83bd-d3f98bda474e\" (UID: \"f5d08644-a56b-4a00-83bd-d3f98bda474e\") "
Sep 10 23:27:48.708812 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.708778    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" (UID: "9463fd1f-3d51-48e3-9c2b-fd75e7f8915c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:48.708812 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.708781    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d08644-a56b-4a00-83bd-d3f98bda474e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5d08644-a56b-4a00-83bd-d3f98bda474e" (UID: "f5d08644-a56b-4a00-83bd-d3f98bda474e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:48.710444 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.710422    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d08644-a56b-4a00-83bd-d3f98bda474e-kube-api-access-6hldm" (OuterVolumeSpecName: "kube-api-access-6hldm") pod "f5d08644-a56b-4a00-83bd-d3f98bda474e" (UID: "f5d08644-a56b-4a00-83bd-d3f98bda474e"). InnerVolumeSpecName "kube-api-access-6hldm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:48.710536 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.710452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-kube-api-access-xjcs4" (OuterVolumeSpecName: "kube-api-access-xjcs4") pod "9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" (UID: "9463fd1f-3d51-48e3-9c2b-fd75e7f8915c"). InnerVolumeSpecName "kube-api-access-xjcs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:48.780799 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.780730    2564 generic.go:356] "Generic (PLEG): container finished" podID="9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" containerID="8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8" exitCode=0
Sep 10 23:27:48.780799 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.780765    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"
Sep 10 23:27:48.780947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.780797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb" event={"ID":"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c","Type":"ContainerDied","Data":"8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8"}
Sep 10 23:27:48.780947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.780851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb" event={"ID":"9463fd1f-3d51-48e3-9c2b-fd75e7f8915c","Type":"ContainerDied","Data":"8854153cf8ea8c6766268a6ee695a60dca175d278b1c3d57372f7e8ada252769"}
Sep 10 23:27:48.780947 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.780873    2564 scope.go:117] "RemoveContainer" containerID="8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8"
Sep 10 23:27:48.782236 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.782216    2564 generic.go:356] "Generic (PLEG): container finished" podID="f5d08644-a56b-4a00-83bd-d3f98bda474e" containerID="d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a" exitCode=0
Sep 10 23:27:48.782340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.782251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"
Sep 10 23:27:48.782340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.782288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6" event={"ID":"f5d08644-a56b-4a00-83bd-d3f98bda474e","Type":"ContainerDied","Data":"d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a"}
Sep 10 23:27:48.782340 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.782327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6" event={"ID":"f5d08644-a56b-4a00-83bd-d3f98bda474e","Type":"ContainerDied","Data":"dac28e9cf405b3d6572e42aa799703176f6b011899277705fbd0dc14582b1e46"}
Sep 10 23:27:48.791573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.791551    2564 scope.go:117] "RemoveContainer" containerID="8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8"
Sep 10 23:27:48.791903 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:27:48.791881    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8\": container with ID starting with 8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8 not found: ID does not exist" containerID="8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8"
Sep 10 23:27:48.791982 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.791908    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8"} err="failed to get container status \"8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8\": rpc error: code = NotFound desc = could not find container \"8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8\": container with ID starting with 8e738a57f579239d80e90dc82de1a7c4e0dce9377dc450b79f1546b80458f5f8 not found: ID does not exist"
Sep 10 23:27:48.791982 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.791924    2564 scope.go:117] "RemoveContainer" containerID="d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a"
Sep 10 23:27:48.800374 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.800357    2564 scope.go:117] "RemoveContainer" containerID="d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a"
Sep 10 23:27:48.800639 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:27:48.800624    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a\": container with ID starting with d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a not found: ID does not exist" containerID="d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a"
Sep 10 23:27:48.800762 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.800686    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a"} err="failed to get container status \"d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a\": rpc error: code = NotFound desc = could not find container \"d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a\": container with ID starting with d3553cb824e6c228632bdf70443af0d40e9a5894d7aeb91fa7be8a9fc64f996a not found: ID does not exist"
Sep 10 23:27:48.803159 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.803140    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"]
Sep 10 23:27:48.809092 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.809066    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d08644-a56b-4a00-83bd-d3f98bda474e-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.809092 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.809086    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjcs4\" (UniqueName: \"kubernetes.io/projected/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-kube-api-access-xjcs4\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.809211 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.809097    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hldm\" (UniqueName: \"kubernetes.io/projected/f5d08644-a56b-4a00-83bd-d3f98bda474e-kube-api-access-6hldm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.809211 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.809105    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:27:48.809879 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.809861    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bda598-58f8675fdc-qqnbb"]
Sep 10 23:27:48.820285 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.820265    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"]
Sep 10 23:27:48.824121 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:48.824097    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bda598-688599fd66-cw6t6"]
Sep 10 23:27:50.198635 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:50.198604    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" path="/var/lib/kubelet/pods/9463fd1f-3d51-48e3-9c2b-fd75e7f8915c/volumes"
Sep 10 23:27:50.199039 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:27:50.198993    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d08644-a56b-4a00-83bd-d3f98bda474e" path="/var/lib/kubelet/pods/f5d08644-a56b-4a00-83bd-d3f98bda474e/volumes"
Sep 10 23:28:06.125659 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.125622    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"]
Sep 10 23:28:06.126251 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.125924    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" podUID="adb995bd-999b-4213-ba03-10e2a0da3198" containerName="maas-api" containerID="cri-o://5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3" gracePeriod=30
Sep 10 23:28:06.174957 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.174927    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"]
Sep 10 23:28:06.175186 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.175143    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh" podUID="72ad6548-b3ac-4fec-bdca-8055f06378c0" containerName="payload-processing" containerID="cri-o://1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c" gracePeriod=30
Sep 10 23:28:06.426307 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.426282    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:28:06.453561 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.453531    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb995bd-999b-4213-ba03-10e2a0da3198-openshift-service-ca\") pod \"adb995bd-999b-4213-ba03-10e2a0da3198\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") "
Sep 10 23:28:06.453699 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.453621    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls\") pod \"adb995bd-999b-4213-ba03-10e2a0da3198\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") "
Sep 10 23:28:06.453699 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.453677    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-metrics-certs\") pod \"adb995bd-999b-4213-ba03-10e2a0da3198\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") "
Sep 10 23:28:06.453851 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.453719    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-574bm\" (UniqueName: \"kubernetes.io/projected/adb995bd-999b-4213-ba03-10e2a0da3198-kube-api-access-574bm\") pod \"adb995bd-999b-4213-ba03-10e2a0da3198\" (UID: \"adb995bd-999b-4213-ba03-10e2a0da3198\") "
Sep 10 23:28:06.453984 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.453960    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb995bd-999b-4213-ba03-10e2a0da3198-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adb995bd-999b-4213-ba03-10e2a0da3198" (UID: "adb995bd-999b-4213-ba03-10e2a0da3198"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:28:06.454075 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.454031    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb995bd-999b-4213-ba03-10e2a0da3198-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.455878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.455848    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adb995bd-999b-4213-ba03-10e2a0da3198" (UID: "adb995bd-999b-4213-ba03-10e2a0da3198"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:28:06.455878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.455859    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb995bd-999b-4213-ba03-10e2a0da3198-kube-api-access-574bm" (OuterVolumeSpecName: "kube-api-access-574bm") pod "adb995bd-999b-4213-ba03-10e2a0da3198" (UID: "adb995bd-999b-4213-ba03-10e2a0da3198"). InnerVolumeSpecName "kube-api-access-574bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:06.456038 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.456004    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adb995bd-999b-4213-ba03-10e2a0da3198" (UID: "adb995bd-999b-4213-ba03-10e2a0da3198"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:28:06.456785 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.456767    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:28:06.554822 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.554792    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ad6548-b3ac-4fec-bdca-8055f06378c0-plugins-config-volume\") pod \"72ad6548-b3ac-4fec-bdca-8055f06378c0\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") "
Sep 10 23:28:06.555007 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.554914    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztm8s\" (UniqueName: \"kubernetes.io/projected/72ad6548-b3ac-4fec-bdca-8055f06378c0-kube-api-access-ztm8s\") pod \"72ad6548-b3ac-4fec-bdca-8055f06378c0\" (UID: \"72ad6548-b3ac-4fec-bdca-8055f06378c0\") "
Sep 10 23:28:06.555183 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.555159    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ad6548-b3ac-4fec-bdca-8055f06378c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72ad6548-b3ac-4fec-bdca-8055f06378c0" (UID: "72ad6548-b3ac-4fec-bdca-8055f06378c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:28:06.555240 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.555169    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.555240 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.555209    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb995bd-999b-4213-ba03-10e2a0da3198-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.555240 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.555222    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574bm\" (UniqueName: \"kubernetes.io/projected/adb995bd-999b-4213-ba03-10e2a0da3198-kube-api-access-574bm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.556905 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.556886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ad6548-b3ac-4fec-bdca-8055f06378c0-kube-api-access-ztm8s" (OuterVolumeSpecName: "kube-api-access-ztm8s") pod "72ad6548-b3ac-4fec-bdca-8055f06378c0" (UID: "72ad6548-b3ac-4fec-bdca-8055f06378c0"). InnerVolumeSpecName "kube-api-access-ztm8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:06.656622 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.656557    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ad6548-b3ac-4fec-bdca-8055f06378c0-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.656622 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.656578    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztm8s\" (UniqueName: \"kubernetes.io/projected/72ad6548-b3ac-4fec-bdca-8055f06378c0-kube-api-access-ztm8s\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.877837 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.877805    2564 generic.go:356] "Generic (PLEG): container finished" podID="adb995bd-999b-4213-ba03-10e2a0da3198" containerID="5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3" exitCode=0
Sep 10 23:28:06.877974 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.877850    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"
Sep 10 23:28:06.877974 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.877888    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" event={"ID":"adb995bd-999b-4213-ba03-10e2a0da3198","Type":"ContainerDied","Data":"5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3"}
Sep 10 23:28:06.877974 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.877945    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq" event={"ID":"adb995bd-999b-4213-ba03-10e2a0da3198","Type":"ContainerDied","Data":"ecddc2eaf3d13627a32c928ef2683b96ff2591936ddac97c34e1624a1adae77c"}
Sep 10 23:28:06.877974 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.877968    2564 scope.go:117] "RemoveContainer" containerID="5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3"
Sep 10 23:28:06.879365 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.879340    2564 generic.go:356] "Generic (PLEG): container finished" podID="72ad6548-b3ac-4fec-bdca-8055f06378c0" containerID="1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c" exitCode=0
Sep 10 23:28:06.879480 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.879387    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"
Sep 10 23:28:06.879480 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.879436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh" event={"ID":"72ad6548-b3ac-4fec-bdca-8055f06378c0","Type":"ContainerDied","Data":"1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c"}
Sep 10 23:28:06.879586 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.879479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh" event={"ID":"72ad6548-b3ac-4fec-bdca-8055f06378c0","Type":"ContainerDied","Data":"e290248ddb9d0c6d8c9889bc4880001de495518d518fabdc3d54cf7dd87bb75a"}
Sep 10 23:28:06.888241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.888225    2564 scope.go:117] "RemoveContainer" containerID="5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3"
Sep 10 23:28:06.888570 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:28:06.888520    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3\": container with ID starting with 5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3 not found: ID does not exist" containerID="5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3"
Sep 10 23:28:06.888570 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.888551    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3"} err="failed to get container status \"5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3\": rpc error: code = NotFound desc = could not find container \"5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3\": container with ID starting with 5f5ac5fafb628f55c0a8252e92d31859d6ee10e2cca46f2d464532ffa8de6fd3 not found: ID does not exist"
Sep 10 23:28:06.888735 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.888574    2564 scope.go:117] "RemoveContainer" containerID="1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c"
Sep 10 23:28:06.897999 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.897981    2564 scope.go:117] "RemoveContainer" containerID="1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c"
Sep 10 23:28:06.898269 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:28:06.898249    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c\": container with ID starting with 1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c not found: ID does not exist" containerID="1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c"
Sep 10 23:28:06.898334 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.898274    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c"} err="failed to get container status \"1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c\": rpc error: code = NotFound desc = could not find container \"1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c\": container with ID starting with 1f370a6fa0883f5343fce9ba1ad3d4ff87bdd1f452aa14534b65e4fa0964d95c not found: ID does not exist"
Sep 10 23:28:06.904551 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.904527    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"]
Sep 10 23:28:06.906553 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.906535    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7b202f-9f7687869-x2dxh"]
Sep 10 23:28:06.914674 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.914627    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"]
Sep 10 23:28:06.920390 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:06.920370    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7b202f-6df6586c47-kdqwq"]
Sep 10 23:28:08.199867 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:08.199835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ad6548-b3ac-4fec-bdca-8055f06378c0" path="/var/lib/kubelet/pods/72ad6548-b3ac-4fec-bdca-8055f06378c0/volumes"
Sep 10 23:28:08.200252 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:08.200235    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb995bd-999b-4213-ba03-10e2a0da3198" path="/var/lib/kubelet/pods/adb995bd-999b-4213-ba03-10e2a0da3198/volumes"
Sep 10 23:28:22.736162 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:22.736125    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"]
Sep 10 23:28:37.756688 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:37.756643    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t" podUID="48dcdfb6-f950-48ee-ab5d-8f10d9478c10" containerName="main" containerID="cri-o://5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85" gracePeriod=45
Sep 10 23:28:38.012800 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.012724    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:28:38.036739 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.036695    2564 generic.go:356] "Generic (PLEG): container finished" podID="48dcdfb6-f950-48ee-ab5d-8f10d9478c10" containerID="5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85" exitCode=0
Sep 10 23:28:38.036896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.036736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t" event={"ID":"48dcdfb6-f950-48ee-ab5d-8f10d9478c10","Type":"ContainerDied","Data":"5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85"}
Sep 10 23:28:38.036896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.036781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t" event={"ID":"48dcdfb6-f950-48ee-ab5d-8f10d9478c10","Type":"ContainerDied","Data":"847fd197b48af41f30a83e2ec715c7bef24e01f30a18728f9d90b45c689d41c2"}
Sep 10 23:28:38.036896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.036800    2564 scope.go:117] "RemoveContainer" containerID="5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85"
Sep 10 23:28:38.036896 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.036744    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"
Sep 10 23:28:38.045137 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.045116    2564 scope.go:117] "RemoveContainer" containerID="5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85"
Sep 10 23:28:38.045411 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:28:38.045385    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85\": container with ID starting with 5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85 not found: ID does not exist" containerID="5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85"
Sep 10 23:28:38.045514 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.045418    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85"} err="failed to get container status \"5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85\": rpc error: code = NotFound desc = could not find container \"5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85\": container with ID starting with 5d945d365cd5ecb3a8e8596c3eea85f7171c73e21b357669ca7ee1c0923fcc85 not found: ID does not exist"
Sep 10 23:28:38.123408 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123387    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-home\") pod \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") "
Sep 10 23:28:38.123499 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123418    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl7vf\" (UniqueName: \"kubernetes.io/projected/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-kube-api-access-jl7vf\") pod \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") "
Sep 10 23:28:38.123499 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123457    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-dshm\") pod \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") "
Sep 10 23:28:38.123499 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123491    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tls-certs\") pod \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") "
Sep 10 23:28:38.123631 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123546    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tmp-dir\") pod \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") "
Sep 10 23:28:38.123631 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123577    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-model-cache\") pod \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\" (UID: \"48dcdfb6-f950-48ee-ab5d-8f10d9478c10\") "
Sep 10 23:28:38.123631 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-home" (OuterVolumeSpecName: "home") pod "48dcdfb6-f950-48ee-ab5d-8f10d9478c10" (UID: "48dcdfb6-f950-48ee-ab5d-8f10d9478c10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:38.123874 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123850    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48dcdfb6-f950-48ee-ab5d-8f10d9478c10" (UID: "48dcdfb6-f950-48ee-ab5d-8f10d9478c10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:38.123927 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123876    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:38.123927 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.123890    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-model-cache" (OuterVolumeSpecName: "model-cache") pod "48dcdfb6-f950-48ee-ab5d-8f10d9478c10" (UID: "48dcdfb6-f950-48ee-ab5d-8f10d9478c10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:38.125652 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.125632    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-kube-api-access-jl7vf" (OuterVolumeSpecName: "kube-api-access-jl7vf") pod "48dcdfb6-f950-48ee-ab5d-8f10d9478c10" (UID: "48dcdfb6-f950-48ee-ab5d-8f10d9478c10"). InnerVolumeSpecName "kube-api-access-jl7vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:38.125774 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.125655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48dcdfb6-f950-48ee-ab5d-8f10d9478c10" (UID: "48dcdfb6-f950-48ee-ab5d-8f10d9478c10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:28:38.125774 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.125686    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-dshm" (OuterVolumeSpecName: "dshm") pod "48dcdfb6-f950-48ee-ab5d-8f10d9478c10" (UID: "48dcdfb6-f950-48ee-ab5d-8f10d9478c10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:38.224292 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.224266    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:38.224292 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.224292    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:38.224441 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.224302    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:38.224441 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.224310    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:38.224441 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.224319    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7vf\" (UniqueName: \"kubernetes.io/projected/48dcdfb6-f950-48ee-ab5d-8f10d9478c10-kube-api-access-jl7vf\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:28:38.351204 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.351140    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"]
Sep 10 23:28:38.357044 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:38.357021    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/auth-test-model-83be66-kserve-7b9976d947-k495t"]
Sep 10 23:28:40.197934 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:40.197905    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48dcdfb6-f950-48ee-ab5d-8f10d9478c10" path="/var/lib/kubelet/pods/48dcdfb6-f950-48ee-ab5d-8f10d9478c10/volumes"
Sep 10 23:28:52.613427 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:28:52.613393    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"]
Sep 10 23:29:03.205072 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.205029    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"]
Sep 10 23:29:03.205551 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.205362    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" podUID="c7b65941-f0b7-4951-a28a-2f617da8b32e" containerName="maas-api" containerID="cri-o://0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f" gracePeriod=30
Sep 10 23:29:03.462090 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.462037    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:29:03.526874 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.526841    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls\") pod \"c7b65941-f0b7-4951-a28a-2f617da8b32e\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") "
Sep 10 23:29:03.527041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.526919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b65941-f0b7-4951-a28a-2f617da8b32e-openshift-service-ca\") pod \"c7b65941-f0b7-4951-a28a-2f617da8b32e\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") "
Sep 10 23:29:03.527041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.526978    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbgln\" (UniqueName: \"kubernetes.io/projected/c7b65941-f0b7-4951-a28a-2f617da8b32e-kube-api-access-lbgln\") pod \"c7b65941-f0b7-4951-a28a-2f617da8b32e\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") "
Sep 10 23:29:03.527041 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.527030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-metrics-certs\") pod \"c7b65941-f0b7-4951-a28a-2f617da8b32e\" (UID: \"c7b65941-f0b7-4951-a28a-2f617da8b32e\") "
Sep 10 23:29:03.527349 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.527320    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b65941-f0b7-4951-a28a-2f617da8b32e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7b65941-f0b7-4951-a28a-2f617da8b32e" (UID: "c7b65941-f0b7-4951-a28a-2f617da8b32e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:03.529008 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.528985    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7b65941-f0b7-4951-a28a-2f617da8b32e" (UID: "c7b65941-f0b7-4951-a28a-2f617da8b32e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:03.529108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.529028    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b65941-f0b7-4951-a28a-2f617da8b32e-kube-api-access-lbgln" (OuterVolumeSpecName: "kube-api-access-lbgln") pod "c7b65941-f0b7-4951-a28a-2f617da8b32e" (UID: "c7b65941-f0b7-4951-a28a-2f617da8b32e"). InnerVolumeSpecName "kube-api-access-lbgln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:03.529163 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.529097    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7b65941-f0b7-4951-a28a-2f617da8b32e" (UID: "c7b65941-f0b7-4951-a28a-2f617da8b32e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:03.628784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.628753    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.628784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.628778    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7b65941-f0b7-4951-a28a-2f617da8b32e-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.628784 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.628788    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b65941-f0b7-4951-a28a-2f617da8b32e-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.628998 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:03.628798    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbgln\" (UniqueName: \"kubernetes.io/projected/c7b65941-f0b7-4951-a28a-2f617da8b32e-kube-api-access-lbgln\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:04.174173 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.174133    2564 generic.go:356] "Generic (PLEG): container finished" podID="c7b65941-f0b7-4951-a28a-2f617da8b32e" containerID="0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f" exitCode=0
Sep 10 23:29:04.174342 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.174184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" event={"ID":"c7b65941-f0b7-4951-a28a-2f617da8b32e","Type":"ContainerDied","Data":"0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f"}
Sep 10 23:29:04.174342 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.174225    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t" event={"ID":"c7b65941-f0b7-4951-a28a-2f617da8b32e","Type":"ContainerDied","Data":"5fcf24dd8ed6621690ff9ff7e63fa2fe32f23e2e969a9b487295d12bc2b6b48f"}
Sep 10 23:29:04.174342 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.174247    2564 scope.go:117] "RemoveContainer" containerID="0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f"
Sep 10 23:29:04.174342 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.174183    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"
Sep 10 23:29:04.183466 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.183447    2564 scope.go:117] "RemoveContainer" containerID="0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f"
Sep 10 23:29:04.183742 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:29:04.183721    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f\": container with ID starting with 0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f not found: ID does not exist" containerID="0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f"
Sep 10 23:29:04.183790 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.183751    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f"} err="failed to get container status \"0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f\": rpc error: code = NotFound desc = could not find container \"0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f\": container with ID starting with 0f0d03f3d15a54f8ff68e2f18f6627203a11990ff658d8dc18e80bdc60949b8f not found: ID does not exist"
Sep 10 23:29:04.199320 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.199301    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"]
Sep 10 23:29:04.200880 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:04.200862    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7025b9-c6d7bff5c-8fw6t"]
Sep 10 23:29:06.198113 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:06.198083    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b65941-f0b7-4951-a28a-2f617da8b32e" path="/var/lib/kubelet/pods/c7b65941-f0b7-4951-a28a-2f617da8b32e/volumes"
Sep 10 23:29:07.632466 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.632429    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d" podUID="552743e7-bc73-4d0b-b705-575c3e521a6e" containerName="main" containerID="cri-o://6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e" gracePeriod=45
Sep 10 23:29:07.880404 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.880380    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:29:07.966013 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.965986    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-home\") pod \"552743e7-bc73-4d0b-b705-575c3e521a6e\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") "
Sep 10 23:29:07.966178 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-model-cache\") pod \"552743e7-bc73-4d0b-b705-575c3e521a6e\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") "
Sep 10 23:29:07.966178 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552743e7-bc73-4d0b-b705-575c3e521a6e-tls-certs\") pod \"552743e7-bc73-4d0b-b705-575c3e521a6e\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") "
Sep 10 23:29:07.966314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966232    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-home" (OuterVolumeSpecName: "home") pod "552743e7-bc73-4d0b-b705-575c3e521a6e" (UID: "552743e7-bc73-4d0b-b705-575c3e521a6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.966314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966237    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-dshm\") pod \"552743e7-bc73-4d0b-b705-575c3e521a6e\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") "
Sep 10 23:29:07.966424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966364    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml5c9\" (UniqueName: \"kubernetes.io/projected/552743e7-bc73-4d0b-b705-575c3e521a6e-kube-api-access-ml5c9\") pod \"552743e7-bc73-4d0b-b705-575c3e521a6e\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") "
Sep 10 23:29:07.966424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966383    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "552743e7-bc73-4d0b-b705-575c3e521a6e" (UID: "552743e7-bc73-4d0b-b705-575c3e521a6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.966424 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-tmp-dir\") pod \"552743e7-bc73-4d0b-b705-575c3e521a6e\" (UID: \"552743e7-bc73-4d0b-b705-575c3e521a6e\") "
Sep 10 23:29:07.966594 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966574    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "552743e7-bc73-4d0b-b705-575c3e521a6e" (UID: "552743e7-bc73-4d0b-b705-575c3e521a6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.966752 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966729    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:07.966813 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966758    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:07.966813 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.966771    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:07.968344 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.968326    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552743e7-bc73-4d0b-b705-575c3e521a6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "552743e7-bc73-4d0b-b705-575c3e521a6e" (UID: "552743e7-bc73-4d0b-b705-575c3e521a6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:07.968452 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.968434    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552743e7-bc73-4d0b-b705-575c3e521a6e-kube-api-access-ml5c9" (OuterVolumeSpecName: "kube-api-access-ml5c9") pod "552743e7-bc73-4d0b-b705-575c3e521a6e" (UID: "552743e7-bc73-4d0b-b705-575c3e521a6e"). InnerVolumeSpecName "kube-api-access-ml5c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:07.968628 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:07.968604    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-dshm" (OuterVolumeSpecName: "dshm") pod "552743e7-bc73-4d0b-b705-575c3e521a6e" (UID: "552743e7-bc73-4d0b-b705-575c3e521a6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.067314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.067284    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5c9\" (UniqueName: \"kubernetes.io/projected/552743e7-bc73-4d0b-b705-575c3e521a6e-kube-api-access-ml5c9\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.067314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.067309    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552743e7-bc73-4d0b-b705-575c3e521a6e-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.067314 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.067318    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552743e7-bc73-4d0b-b705-575c3e521a6e-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.196888 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.196853    2564 generic.go:356] "Generic (PLEG): container finished" podID="552743e7-bc73-4d0b-b705-575c3e521a6e" containerID="6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e" exitCode=0
Sep 10 23:29:08.197081 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.196968    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"
Sep 10 23:29:08.197924 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.197901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d" event={"ID":"552743e7-bc73-4d0b-b705-575c3e521a6e","Type":"ContainerDied","Data":"6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e"}
Sep 10 23:29:08.198065 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.197931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d" event={"ID":"552743e7-bc73-4d0b-b705-575c3e521a6e","Type":"ContainerDied","Data":"a2b6b6d39c5d200bec5cf5921e50a3ee342b703a20088cf717b1ab88038e6ccd"}
Sep 10 23:29:08.198065 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.197949    2564 scope.go:117] "RemoveContainer" containerID="6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e"
Sep 10 23:29:08.207645 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.207618    2564 scope.go:117] "RemoveContainer" containerID="6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e"
Sep 10 23:29:08.207925 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:29:08.207905    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e\": container with ID starting with 6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e not found: ID does not exist" containerID="6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e"
Sep 10 23:29:08.208011 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.207937    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e"} err="failed to get container status \"6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e\": rpc error: code = NotFound desc = could not find container \"6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e\": container with ID starting with 6d97633ac826f7676eabcd569bb535eafdb1e3d7fb228351da612f2a0b4cbf3e not found: ID does not exist"
Sep 10 23:29:08.224736 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.224660    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"]
Sep 10 23:29:08.226298 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:08.226279    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/unconfigured-w4-7025b9-kserve-855966678c-xxh2d"]
Sep 10 23:29:10.200261 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:29:10.200229    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552743e7-bc73-4d0b-b705-575c3e521a6e" path="/var/lib/kubelet/pods/552743e7-bc73-4d0b-b705-575c3e521a6e/volumes"
Sep 10 23:30:00.133758 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.133667    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"]
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134061    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a576bf-fe1a-423a-b922-1bb6f2477c0b" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134073    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a576bf-fe1a-423a-b922-1bb6f2477c0b" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134085    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" containerName="payload-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134090    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" containerName="payload-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134100    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b65941-f0b7-4951-a28a-2f617da8b32e" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134105    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b65941-f0b7-4951-a28a-2f617da8b32e" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134112    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d08644-a56b-4a00-83bd-d3f98bda474e" containerName="payload-pre-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134117    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d08644-a56b-4a00-83bd-d3f98bda474e" containerName="payload-pre-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134125    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb995bd-999b-4213-ba03-10e2a0da3198" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134130    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb995bd-999b-4213-ba03-10e2a0da3198" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134140    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ad6548-b3ac-4fec-bdca-8055f06378c0" containerName="payload-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134145    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ad6548-b3ac-4fec-bdca-8055f06378c0" containerName="payload-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134151    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" containerName="payload-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134156    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" containerName="payload-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134164    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88cd0224-3942-4349-a9b6-7625dc50aed0" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134168    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="88cd0224-3942-4349-a9b6-7625dc50aed0" containerName="maas-api"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134174    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552743e7-bc73-4d0b-b705-575c3e521a6e" containerName="main"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134179    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="552743e7-bc73-4d0b-b705-575c3e521a6e" containerName="main"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134188    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dcdfb6-f950-48ee-ab5d-8f10d9478c10" containerName="main"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134194    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dcdfb6-f950-48ee-ab5d-8f10d9478c10" containerName="main"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134200    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887cd0f5-ce74-421e-b641-17f013663d34" containerName="payload-pre-processing"
Sep 10 23:30:00.134199 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134205    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="887cd0f5-ce74-421e-b641-17f013663d34" containerName="payload-pre-processing"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134267    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3ce8d3-a1f8-4f1e-ac55-623fb78313d2" containerName="payload-processing"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134274    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9463fd1f-3d51-48e3-9c2b-fd75e7f8915c" containerName="payload-processing"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134279    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb995bd-999b-4213-ba03-10e2a0da3198" containerName="maas-api"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134285    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="88cd0224-3942-4349-a9b6-7625dc50aed0" containerName="maas-api"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134292    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="552743e7-bc73-4d0b-b705-575c3e521a6e" containerName="main"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134297    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a576bf-fe1a-423a-b922-1bb6f2477c0b" containerName="maas-api"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134302    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="48dcdfb6-f950-48ee-ab5d-8f10d9478c10" containerName="main"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134308    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b65941-f0b7-4951-a28a-2f617da8b32e" containerName="maas-api"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134313    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="887cd0f5-ce74-421e-b641-17f013663d34" containerName="payload-pre-processing"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134319    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d08644-a56b-4a00-83bd-d3f98bda474e" containerName="payload-pre-processing"
Sep 10 23:30:00.134954 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.134323    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ad6548-b3ac-4fec-bdca-8055f06378c0" containerName="payload-processing"
Sep 10 23:30:00.137149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.137133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"
Sep 10 23:30:00.139445 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.139423    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:30:00.147743 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.147722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"
Sep 10 23:30:00.147870 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.147739    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"]
Sep 10 23:30:00.419149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.419108    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"]
Sep 10 23:30:00.421732 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:30:00.421692    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc646f8_5d55_40d6_9d59_0ad635a1a922.slice/crio-de92a7b0c0deef343dad47f2e6e7824a95e8cef700fa6bdf5af8881089ba0720 WatchSource:0}: Error finding container de92a7b0c0deef343dad47f2e6e7824a95e8cef700fa6bdf5af8881089ba0720: Status 404 returned error can't find the container with id de92a7b0c0deef343dad47f2e6e7824a95e8cef700fa6bdf5af8881089ba0720
Sep 10 23:30:00.451574 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:00.451552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95" event={"ID":"0cc646f8-5d55-40d6-9d59-0ad635a1a922","Type":"ContainerStarted","Data":"de92a7b0c0deef343dad47f2e6e7824a95e8cef700fa6bdf5af8881089ba0720"}
Sep 10 23:30:05.478345 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:05.478312    2564 generic.go:356] "Generic (PLEG): container finished" podID="0cc646f8-5d55-40d6-9d59-0ad635a1a922" containerID="4f589d823071ef1f364cf9fd0679ffa7069e8351d043634cfa3b29bef4c50000" exitCode=0
Sep 10 23:30:05.478655 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:05.478353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95" event={"ID":"0cc646f8-5d55-40d6-9d59-0ad635a1a922","Type":"ContainerDied","Data":"4f589d823071ef1f364cf9fd0679ffa7069e8351d043634cfa3b29bef4c50000"}
Sep 10 23:30:06.609756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:06.609714    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"
Sep 10 23:30:07.489471 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:07.489446    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"
Sep 10 23:30:07.489471 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:07.489455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95" event={"ID":"0cc646f8-5d55-40d6-9d59-0ad635a1a922","Type":"ContainerDied","Data":"de92a7b0c0deef343dad47f2e6e7824a95e8cef700fa6bdf5af8881089ba0720"}
Sep 10 23:30:07.489670 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:30:07.489496    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de92a7b0c0deef343dad47f2e6e7824a95e8cef700fa6bdf5af8881089ba0720"
Sep 10 23:31:36.243848 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:36.243817    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:31:36.258758 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:36.258732    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:31:49.143599 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.143567    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"]
Sep 10 23:31:49.144179 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.144041    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc646f8-5d55-40d6-9d59-0ad635a1a922" containerName="cleanup"
Sep 10 23:31:49.144179 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.144067    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc646f8-5d55-40d6-9d59-0ad635a1a922" containerName="cleanup"
Sep 10 23:31:49.144179 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.144176    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc646f8-5d55-40d6-9d59-0ad635a1a922" containerName="cleanup"
Sep 10 23:31:49.147322 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.147298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.149557 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.149536    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ea2b31-serving-cert\""
Sep 10 23:31:49.151469 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.151442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36642c88-849b-49eb-b122-3886e0e88428-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.151601 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.151489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.151601 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.151513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-metrics-certs\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.151687 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.151642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fk6\" (UniqueName: \"kubernetes.io/projected/36642c88-849b-49eb-b122-3886e0e88428-kube-api-access-62fk6\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.159746 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.159724    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"]
Sep 10 23:31:49.252137 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.252109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fk6\" (UniqueName: \"kubernetes.io/projected/36642c88-849b-49eb-b122-3886e0e88428-kube-api-access-62fk6\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.252291 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.252159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36642c88-849b-49eb-b122-3886e0e88428-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.252291 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.252183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.252291 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.252198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-metrics-certs\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.252420 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:31:49.252379    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-serving-cert: secret "maas-api-e2e-ipp-a-ea2b31-serving-cert" not found
Sep 10 23:31:49.252466 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:31:49.252448    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls podName:36642c88-849b-49eb-b122-3886e0e88428 nodeName:}" failed. No retries permitted until 2026-09-10 23:31:49.752429238 +0000 UTC m=+1814.038450437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls") pod "maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" (UID: "36642c88-849b-49eb-b122-3886e0e88428") : secret "maas-api-e2e-ipp-a-ea2b31-serving-cert" not found
Sep 10 23:31:49.252834 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.252814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36642c88-849b-49eb-b122-3886e0e88428-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.254511 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.254494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-metrics-certs\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.278625 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.278599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fk6\" (UniqueName: \"kubernetes.io/projected/36642c88-849b-49eb-b122-3886e0e88428-kube-api-access-62fk6\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.756769 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.756733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.759171 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.759144    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:49.784957 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.784932    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"]
Sep 10 23:31:49.789830 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.789812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.791920 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.791900    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ea2b31-dockercfg-p5rhl\""
Sep 10 23:31:49.792024 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.791903    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ea2b31\""
Sep 10 23:31:49.797951 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.797926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"]
Sep 10 23:31:49.818640 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.818609    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"]
Sep 10 23:31:49.821951 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.821936    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:49.831535 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.831514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"]
Sep 10 23:31:49.857427 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.857395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c6d351-26e7-4fa6-a977-ddb594c193bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:49.857561 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.857440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b6a383-7fff-4218-9aaa-405d4b1d4a23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.857561 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.857507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5b6\" (UniqueName: \"kubernetes.io/projected/79b6a383-7fff-4218-9aaa-405d4b1d4a23-kube-api-access-4w5b6\") pod \"payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.857561 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.857533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qt4\" (UniqueName: \"kubernetes.io/projected/27c6d351-26e7-4fa6-a977-ddb594c193bd-kube-api-access-x9qt4\") pod \"payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:49.958799 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.958766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5b6\" (UniqueName: \"kubernetes.io/projected/79b6a383-7fff-4218-9aaa-405d4b1d4a23-kube-api-access-4w5b6\") pod \"payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.958943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.958804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qt4\" (UniqueName: \"kubernetes.io/projected/27c6d351-26e7-4fa6-a977-ddb594c193bd-kube-api-access-x9qt4\") pod \"payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:49.958943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.958853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c6d351-26e7-4fa6-a977-ddb594c193bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:49.958943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.958882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b6a383-7fff-4218-9aaa-405d4b1d4a23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.959792 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.959763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b6a383-7fff-4218-9aaa-405d4b1d4a23-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.959904 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.959763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c6d351-26e7-4fa6-a977-ddb594c193bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:49.966327 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.966303    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5b6\" (UniqueName: \"kubernetes.io/projected/79b6a383-7fff-4218-9aaa-405d4b1d4a23-kube-api-access-4w5b6\") pod \"payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:49.966476 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:49.966452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qt4\" (UniqueName: \"kubernetes.io/projected/27c6d351-26e7-4fa6-a977-ddb594c193bd-kube-api-access-x9qt4\") pod \"payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:50.059174 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.059097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:50.101863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.101829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:50.131952 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.131922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:50.328927 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.328898    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"]
Sep 10 23:31:50.330947 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:31:50.330922    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36642c88_849b_49eb_b122_3886e0e88428.slice/crio-9eb869cc4a1ed9efc6123faa28a89e5aed66cc0b706bf12ad24ef600302448a9 WatchSource:0}: Error finding container 9eb869cc4a1ed9efc6123faa28a89e5aed66cc0b706bf12ad24ef600302448a9: Status 404 returned error can't find the container with id 9eb869cc4a1ed9efc6123faa28a89e5aed66cc0b706bf12ad24ef600302448a9
Sep 10 23:31:50.332510 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.332486    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:31:50.388743 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.388719    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"]
Sep 10 23:31:50.391257 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:31:50.391235    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6a383_7fff_4218_9aaa_405d4b1d4a23.slice/crio-e89717bafd92845f8ecca58ffb427699903e7ad3a9178cb395080c5dc952f98b WatchSource:0}: Error finding container e89717bafd92845f8ecca58ffb427699903e7ad3a9178cb395080c5dc952f98b: Status 404 returned error can't find the container with id e89717bafd92845f8ecca58ffb427699903e7ad3a9178cb395080c5dc952f98b
Sep 10 23:31:50.417885 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.417860    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"]
Sep 10 23:31:50.428911 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:31:50.428871    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c6d351_26e7_4fa6_a977_ddb594c193bd.slice/crio-43997d4aadefaca3724e42af3efe7784bf74f143a77e76119abf82c7f6f2521a WatchSource:0}: Error finding container 43997d4aadefaca3724e42af3efe7784bf74f143a77e76119abf82c7f6f2521a: Status 404 returned error can't find the container with id 43997d4aadefaca3724e42af3efe7784bf74f143a77e76119abf82c7f6f2521a
Sep 10 23:31:50.988778 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.988746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v" event={"ID":"79b6a383-7fff-4218-9aaa-405d4b1d4a23","Type":"ContainerStarted","Data":"ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87"}
Sep 10 23:31:50.988936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.988807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v" event={"ID":"79b6a383-7fff-4218-9aaa-405d4b1d4a23","Type":"ContainerStarted","Data":"e89717bafd92845f8ecca58ffb427699903e7ad3a9178cb395080c5dc952f98b"}
Sep 10 23:31:50.988936 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.988855    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:31:50.990738 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.990633    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5" event={"ID":"27c6d351-26e7-4fa6-a977-ddb594c193bd","Type":"ContainerStarted","Data":"47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9"}
Sep 10 23:31:50.990738 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.990683    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5" event={"ID":"27c6d351-26e7-4fa6-a977-ddb594c193bd","Type":"ContainerStarted","Data":"43997d4aadefaca3724e42af3efe7784bf74f143a77e76119abf82c7f6f2521a"}
Sep 10 23:31:50.990738 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.990728    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:31:50.992590 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.992565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" event={"ID":"36642c88-849b-49eb-b122-3886e0e88428","Type":"ContainerStarted","Data":"0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b"}
Sep 10 23:31:50.992734 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.992607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" event={"ID":"36642c88-849b-49eb-b122-3886e0e88428","Type":"ContainerStarted","Data":"9eb869cc4a1ed9efc6123faa28a89e5aed66cc0b706bf12ad24ef600302448a9"}
Sep 10 23:31:50.992734 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:50.992647    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:31:51.005540 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:51.005351    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v" podStartSLOduration=2.005336847 podStartE2EDuration="2.005336847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:51.004630297 +0000 UTC m=+1815.290651499" watchObservedRunningTime="2026-09-10 23:31:51.005336847 +0000 UTC m=+1815.291358048"
Sep 10 23:31:51.022188 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:51.022137    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5" podStartSLOduration=2.022126813 podStartE2EDuration="2.022126813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:51.020040431 +0000 UTC m=+1815.306061638" watchObservedRunningTime="2026-09-10 23:31:51.022126813 +0000 UTC m=+1815.308148014"
Sep 10 23:31:51.036482 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:51.036446    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" podStartSLOduration=1.57939104 podStartE2EDuration="2.036433208s" totalImagesPullingTime="457.042168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:51.034646655 +0000 UTC m=+1815.320667857" watchObservedRunningTime="2026-09-10 23:31:51.036433208 +0000 UTC m=+1815.322454409"
Sep 10 23:31:57.005991 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:31:57.005960    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:32:02.001665 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:02.001633    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:32:02.002059 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:02.001731    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:32:18.259009 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.258970    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"]
Sep 10 23:32:18.263095 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.263068    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.265356 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.265335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-706b9d-serving-cert\""
Sep 10 23:32:18.271330 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.271301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"]
Sep 10 23:32:18.389697 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.389657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm276\" (UniqueName: \"kubernetes.io/projected/ad02d0e6-0395-446e-85b0-31b307bc7138-kube-api-access-lm276\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.389878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.389739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02d0e6-0395-446e-85b0-31b307bc7138-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.389878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.389766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-metrics-certs\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.389878 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.389794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.490226 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.490195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02d0e6-0395-446e-85b0-31b307bc7138-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.490226 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.490229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-metrics-certs\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.490448 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.490252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.490448 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.490324    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm276\" (UniqueName: \"kubernetes.io/projected/ad02d0e6-0395-446e-85b0-31b307bc7138-kube-api-access-lm276\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.490448 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:32:18.490416    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-serving-cert: secret "maas-api-e2e-ipp-b-706b9d-serving-cert" not found
Sep 10 23:32:18.490609 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:32:18.490488    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls podName:ad02d0e6-0395-446e-85b0-31b307bc7138 nodeName:}" failed. No retries permitted until 2026-09-10 23:32:18.990473179 +0000 UTC m=+1843.276494358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls") pod "maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" (UID: "ad02d0e6-0395-446e-85b0-31b307bc7138") : secret "maas-api-e2e-ipp-b-706b9d-serving-cert" not found
Sep 10 23:32:18.490869 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.490847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02d0e6-0395-446e-85b0-31b307bc7138-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.492962 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.492939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-metrics-certs\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.505902 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.505872    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm276\" (UniqueName: \"kubernetes.io/projected/ad02d0e6-0395-446e-85b0-31b307bc7138-kube-api-access-lm276\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.994388 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.994353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:18.996933 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:18.996908    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls\") pod \"maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:19.176221 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:19.176185    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:19.454730 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:19.449621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"]
Sep 10 23:32:19.456769 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:32:19.456739    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad02d0e6_0395_446e_85b0_31b307bc7138.slice/crio-be62ffcc4ff211dae561fa3c6ce2f1f926f093dd0df04decef7fb34ce1d4aa80 WatchSource:0}: Error finding container be62ffcc4ff211dae561fa3c6ce2f1f926f093dd0df04decef7fb34ce1d4aa80: Status 404 returned error can't find the container with id be62ffcc4ff211dae561fa3c6ce2f1f926f093dd0df04decef7fb34ce1d4aa80
Sep 10 23:32:20.141312 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:20.141229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" event={"ID":"ad02d0e6-0395-446e-85b0-31b307bc7138","Type":"ContainerStarted","Data":"584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165"}
Sep 10 23:32:20.141312 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:20.141286    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" event={"ID":"ad02d0e6-0395-446e-85b0-31b307bc7138","Type":"ContainerStarted","Data":"be62ffcc4ff211dae561fa3c6ce2f1f926f093dd0df04decef7fb34ce1d4aa80"}
Sep 10 23:32:20.141638 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:20.141315    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:32:20.157965 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:20.157918    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" podStartSLOduration=1.841511965 podStartE2EDuration="2.157903707s" totalImagesPullingTime="316.391742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:32:20.156094976 +0000 UTC m=+1844.442116178" watchObservedRunningTime="2026-09-10 23:32:20.157903707 +0000 UTC m=+1844.443924949"
Sep 10 23:32:26.151663 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:32:26.151629    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:33:17.641073 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:33:17.641036    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5_27c6d351-26e7-4fa6-a977-ddb594c193bd/payload-processing/0.log"
Sep 10 23:34:19.079876 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.079840    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"]
Sep 10 23:34:19.080394 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.080104    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" podUID="36642c88-849b-49eb-b122-3886e0e88428" containerName="maas-api" containerID="cri-o://0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b" gracePeriod=30
Sep 10 23:34:19.122746 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.122716    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"]
Sep 10 23:34:19.122961 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.122936    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5" podUID="27c6d351-26e7-4fa6-a977-ddb594c193bd" containerName="payload-processing" containerID="cri-o://47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9" gracePeriod=30
Sep 10 23:34:19.128601 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.128578    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"]
Sep 10 23:34:19.128930 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.128902    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v" podUID="79b6a383-7fff-4218-9aaa-405d4b1d4a23" containerName="payload-pre-processing" containerID="cri-o://ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87" gracePeriod=30
Sep 10 23:34:19.405641 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.405613    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:34:19.431553 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.431524    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:34:19.434811 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.434792    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:34:19.488503 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488474    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36642c88-849b-49eb-b122-3886e0e88428-openshift-service-ca\") pod \"36642c88-849b-49eb-b122-3886e0e88428\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") "
Sep 10 23:34:19.488503 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488503    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9qt4\" (UniqueName: \"kubernetes.io/projected/27c6d351-26e7-4fa6-a977-ddb594c193bd-kube-api-access-x9qt4\") pod \"27c6d351-26e7-4fa6-a977-ddb594c193bd\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") "
Sep 10 23:34:19.488677 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488548    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62fk6\" (UniqueName: \"kubernetes.io/projected/36642c88-849b-49eb-b122-3886e0e88428-kube-api-access-62fk6\") pod \"36642c88-849b-49eb-b122-3886e0e88428\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") "
Sep 10 23:34:19.488677 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c6d351-26e7-4fa6-a977-ddb594c193bd-plugins-config-volume\") pod \"27c6d351-26e7-4fa6-a977-ddb594c193bd\" (UID: \"27c6d351-26e7-4fa6-a977-ddb594c193bd\") "
Sep 10 23:34:19.488787 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488684    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w5b6\" (UniqueName: \"kubernetes.io/projected/79b6a383-7fff-4218-9aaa-405d4b1d4a23-kube-api-access-4w5b6\") pod \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") "
Sep 10 23:34:19.488787 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488759    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b6a383-7fff-4218-9aaa-405d4b1d4a23-plugins-config-volume\") pod \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\" (UID: \"79b6a383-7fff-4218-9aaa-405d4b1d4a23\") "
Sep 10 23:34:19.488884 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls\") pod \"36642c88-849b-49eb-b122-3886e0e88428\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") "
Sep 10 23:34:19.488884 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.488837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-metrics-certs\") pod \"36642c88-849b-49eb-b122-3886e0e88428\" (UID: \"36642c88-849b-49eb-b122-3886e0e88428\") "
Sep 10 23:34:19.489028 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.489002    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c6d351-26e7-4fa6-a977-ddb594c193bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27c6d351-26e7-4fa6-a977-ddb594c193bd" (UID: "27c6d351-26e7-4fa6-a977-ddb594c193bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:34:19.489107 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.489022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36642c88-849b-49eb-b122-3886e0e88428-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36642c88-849b-49eb-b122-3886e0e88428" (UID: "36642c88-849b-49eb-b122-3886e0e88428"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:34:19.489241 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.489222    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c6d351-26e7-4fa6-a977-ddb594c193bd-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.489322 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.489246    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36642c88-849b-49eb-b122-3886e0e88428-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.489497 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.489469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6a383-7fff-4218-9aaa-405d4b1d4a23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79b6a383-7fff-4218-9aaa-405d4b1d4a23" (UID: "79b6a383-7fff-4218-9aaa-405d4b1d4a23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:34:19.491000 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.490964    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36642c88-849b-49eb-b122-3886e0e88428" (UID: "36642c88-849b-49eb-b122-3886e0e88428"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:34:19.491000 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.490980    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36642c88-849b-49eb-b122-3886e0e88428" (UID: "36642c88-849b-49eb-b122-3886e0e88428"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:34:19.491149 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.491067    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6a383-7fff-4218-9aaa-405d4b1d4a23-kube-api-access-4w5b6" (OuterVolumeSpecName: "kube-api-access-4w5b6") pod "79b6a383-7fff-4218-9aaa-405d4b1d4a23" (UID: "79b6a383-7fff-4218-9aaa-405d4b1d4a23"). InnerVolumeSpecName "kube-api-access-4w5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:34:19.491202 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.491184    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36642c88-849b-49eb-b122-3886e0e88428-kube-api-access-62fk6" (OuterVolumeSpecName: "kube-api-access-62fk6") pod "36642c88-849b-49eb-b122-3886e0e88428" (UID: "36642c88-849b-49eb-b122-3886e0e88428"). InnerVolumeSpecName "kube-api-access-62fk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:34:19.491240 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.491219    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c6d351-26e7-4fa6-a977-ddb594c193bd-kube-api-access-x9qt4" (OuterVolumeSpecName: "kube-api-access-x9qt4") pod "27c6d351-26e7-4fa6-a977-ddb594c193bd" (UID: "27c6d351-26e7-4fa6-a977-ddb594c193bd"). InnerVolumeSpecName "kube-api-access-x9qt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:34:19.590585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.590528    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b6a383-7fff-4218-9aaa-405d4b1d4a23-plugins-config-volume\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.590585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.590551    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.590585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.590560    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36642c88-849b-49eb-b122-3886e0e88428-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.590585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.590568    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9qt4\" (UniqueName: \"kubernetes.io/projected/27c6d351-26e7-4fa6-a977-ddb594c193bd-kube-api-access-x9qt4\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.590585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.590576    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62fk6\" (UniqueName: \"kubernetes.io/projected/36642c88-849b-49eb-b122-3886e0e88428-kube-api-access-62fk6\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.590585 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.590584    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5b6\" (UniqueName: \"kubernetes.io/projected/79b6a383-7fff-4218-9aaa-405d4b1d4a23-kube-api-access-4w5b6\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:34:19.744097 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.744054    2564 generic.go:356] "Generic (PLEG): container finished" podID="79b6a383-7fff-4218-9aaa-405d4b1d4a23" containerID="ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87" exitCode=0
Sep 10 23:34:19.744249 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.744114    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"
Sep 10 23:34:19.744249 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.744128    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v" event={"ID":"79b6a383-7fff-4218-9aaa-405d4b1d4a23","Type":"ContainerDied","Data":"ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87"}
Sep 10 23:34:19.744249 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.744222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v" event={"ID":"79b6a383-7fff-4218-9aaa-405d4b1d4a23","Type":"ContainerDied","Data":"e89717bafd92845f8ecca58ffb427699903e7ad3a9178cb395080c5dc952f98b"}
Sep 10 23:34:19.744392 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.744259    2564 scope.go:117] "RemoveContainer" containerID="ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87"
Sep 10 23:34:19.745555 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.745523    2564 generic.go:356] "Generic (PLEG): container finished" podID="27c6d351-26e7-4fa6-a977-ddb594c193bd" containerID="47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9" exitCode=0
Sep 10 23:34:19.745679 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.745559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5" event={"ID":"27c6d351-26e7-4fa6-a977-ddb594c193bd","Type":"ContainerDied","Data":"47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9"}
Sep 10 23:34:19.745679 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.745590    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"
Sep 10 23:34:19.745679 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.745607    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5" event={"ID":"27c6d351-26e7-4fa6-a977-ddb594c193bd","Type":"ContainerDied","Data":"43997d4aadefaca3724e42af3efe7784bf74f143a77e76119abf82c7f6f2521a"}
Sep 10 23:34:19.747134 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.747109    2564 generic.go:356] "Generic (PLEG): container finished" podID="36642c88-849b-49eb-b122-3886e0e88428" containerID="0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b" exitCode=0
Sep 10 23:34:19.747317 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.747164    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"
Sep 10 23:34:19.747317 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.747178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" event={"ID":"36642c88-849b-49eb-b122-3886e0e88428","Type":"ContainerDied","Data":"0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b"}
Sep 10 23:34:19.747317 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.747215    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr" event={"ID":"36642c88-849b-49eb-b122-3886e0e88428","Type":"ContainerDied","Data":"9eb869cc4a1ed9efc6123faa28a89e5aed66cc0b706bf12ad24ef600302448a9"}
Sep 10 23:34:19.755866 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.755848    2564 scope.go:117] "RemoveContainer" containerID="ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87"
Sep 10 23:34:19.756151 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:34:19.756127    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87\": container with ID starting with ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87 not found: ID does not exist" containerID="ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87"
Sep 10 23:34:19.756212 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.756164    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87"} err="failed to get container status \"ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87\": rpc error: code = NotFound desc = could not find container \"ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87\": container with ID starting with ccbf567e1c492ee70708e20181e8d9819cf6d7225135e4f470d080c6e3630f87 not found: ID does not exist"
Sep 10 23:34:19.756212 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.756186    2564 scope.go:117] "RemoveContainer" containerID="47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9"
Sep 10 23:34:19.764271 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.764253    2564 scope.go:117] "RemoveContainer" containerID="47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9"
Sep 10 23:34:19.764517 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:34:19.764501    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9\": container with ID starting with 47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9 not found: ID does not exist" containerID="47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9"
Sep 10 23:34:19.764575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.764522    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9"} err="failed to get container status \"47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9\": rpc error: code = NotFound desc = could not find container \"47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9\": container with ID starting with 47f67d0a0f8e8ea7cffda1f1263baef75eef7ca8adde5ae9c2f8732a48e7ccf9 not found: ID does not exist"
Sep 10 23:34:19.764575 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.764536    2564 scope.go:117] "RemoveContainer" containerID="0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b"
Sep 10 23:34:19.771629 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.771603    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"]
Sep 10 23:34:19.772487 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.772475    2564 scope.go:117] "RemoveContainer" containerID="0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b"
Sep 10 23:34:19.772890 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:34:19.772862    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b\": container with ID starting with 0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b not found: ID does not exist" containerID="0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b"
Sep 10 23:34:19.772997 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.772896    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b"} err="failed to get container status \"0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b\": rpc error: code = NotFound desc = could not find container \"0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b\": container with ID starting with 0d5aa8c732ecaa38d47cdf200198667c48bd508c99324bbea6a00b9b2252425b not found: ID does not exist"
Sep 10 23:34:19.775148 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.775128    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ea2b31-7bc7597d5c-8jld5"]
Sep 10 23:34:19.784184 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.784165    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"]
Sep 10 23:34:19.787885 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.787863    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ea2b31-55f6d848c8-zmb7v"]
Sep 10 23:34:19.797869 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.797803    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"]
Sep 10 23:34:19.799869 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:19.799852    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ea2b31-7977ff64b5-ql4lr"]
Sep 10 23:34:20.199015 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:20.198985    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c6d351-26e7-4fa6-a977-ddb594c193bd" path="/var/lib/kubelet/pods/27c6d351-26e7-4fa6-a977-ddb594c193bd/volumes"
Sep 10 23:34:20.199594 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:20.199570    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36642c88-849b-49eb-b122-3886e0e88428" path="/var/lib/kubelet/pods/36642c88-849b-49eb-b122-3886e0e88428/volumes"
Sep 10 23:34:20.199995 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:34:20.199977    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b6a383-7fff-4218-9aaa-405d4b1d4a23" path="/var/lib/kubelet/pods/79b6a383-7fff-4218-9aaa-405d4b1d4a23/volumes"
Sep 10 23:35:18.974373 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:18.974339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"]
Sep 10 23:35:18.974852 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:18.974593    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" podUID="ad02d0e6-0395-446e-85b0-31b307bc7138" containerName="maas-api" containerID="cri-o://584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165" gracePeriod=30
Sep 10 23:35:19.225538 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.225478    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:35:19.373780 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.373740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02d0e6-0395-446e-85b0-31b307bc7138-openshift-service-ca\") pod \"ad02d0e6-0395-446e-85b0-31b307bc7138\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") "
Sep 10 23:35:19.373780 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.373784    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls\") pod \"ad02d0e6-0395-446e-85b0-31b307bc7138\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") "
Sep 10 23:35:19.374025 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.373827    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-metrics-certs\") pod \"ad02d0e6-0395-446e-85b0-31b307bc7138\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") "
Sep 10 23:35:19.374025 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.373863    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm276\" (UniqueName: \"kubernetes.io/projected/ad02d0e6-0395-446e-85b0-31b307bc7138-kube-api-access-lm276\") pod \"ad02d0e6-0395-446e-85b0-31b307bc7138\" (UID: \"ad02d0e6-0395-446e-85b0-31b307bc7138\") "
Sep 10 23:35:19.374154 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.374077    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02d0e6-0395-446e-85b0-31b307bc7138-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad02d0e6-0395-446e-85b0-31b307bc7138" (UID: "ad02d0e6-0395-446e-85b0-31b307bc7138"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:19.374217 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.374198    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02d0e6-0395-446e-85b0-31b307bc7138-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.376106 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.376071    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad02d0e6-0395-446e-85b0-31b307bc7138" (UID: "ad02d0e6-0395-446e-85b0-31b307bc7138"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:19.376232 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.376103    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad02d0e6-0395-446e-85b0-31b307bc7138" (UID: "ad02d0e6-0395-446e-85b0-31b307bc7138"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:19.376381 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.376363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad02d0e6-0395-446e-85b0-31b307bc7138-kube-api-access-lm276" (OuterVolumeSpecName: "kube-api-access-lm276") pod "ad02d0e6-0395-446e-85b0-31b307bc7138" (UID: "ad02d0e6-0395-446e-85b0-31b307bc7138"). InnerVolumeSpecName "kube-api-access-lm276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:19.475260 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.475233    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.475260 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.475256    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02d0e6-0395-446e-85b0-31b307bc7138-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.475260 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:19.475265    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm276\" (UniqueName: \"kubernetes.io/projected/ad02d0e6-0395-446e-85b0-31b307bc7138-kube-api-access-lm276\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:35:20.041355 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.041324    2564 generic.go:356] "Generic (PLEG): container finished" podID="ad02d0e6-0395-446e-85b0-31b307bc7138" containerID="584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165" exitCode=0
Sep 10 23:35:20.041887 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.041375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" event={"ID":"ad02d0e6-0395-446e-85b0-31b307bc7138","Type":"ContainerDied","Data":"584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165"}
Sep 10 23:35:20.041887 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.041380    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"
Sep 10 23:35:20.041887 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.041424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v" event={"ID":"ad02d0e6-0395-446e-85b0-31b307bc7138","Type":"ContainerDied","Data":"be62ffcc4ff211dae561fa3c6ce2f1f926f093dd0df04decef7fb34ce1d4aa80"}
Sep 10 23:35:20.041887 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.041448    2564 scope.go:117] "RemoveContainer" containerID="584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165"
Sep 10 23:35:20.051068 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.051052    2564 scope.go:117] "RemoveContainer" containerID="584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165"
Sep 10 23:35:20.051312 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:35:20.051295    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165\": container with ID starting with 584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165 not found: ID does not exist" containerID="584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165"
Sep 10 23:35:20.051378 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.051319    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165"} err="failed to get container status \"584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165\": rpc error: code = NotFound desc = could not find container \"584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165\": container with ID starting with 584c7476bb995f4ce23c6b59da0552ca2605c2215f65cc22753f348d38371165 not found: ID does not exist"
Sep 10 23:35:20.062372 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.062349    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"]
Sep 10 23:35:20.067819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.067797    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-706b9d-64dc4794c6-xw75v"]
Sep 10 23:35:20.197886 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:20.197862    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad02d0e6-0395-446e-85b0-31b307bc7138" path="/var/lib/kubelet/pods/ad02d0e6-0395-446e-85b0-31b307bc7138/volumes"
Sep 10 23:35:36.728449 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:36.728385    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"]
Sep 10 23:35:36.731857 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:36.731827    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6c97f2-29818050-t2p95"]
Sep 10 23:35:38.197914 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:35:38.197873    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc646f8-5d55-40d6-9d59-0ad635a1a922" path="/var/lib/kubelet/pods/0cc646f8-5d55-40d6-9d59-0ad635a1a922/volumes"
Sep 10 23:36:03.771002 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:03.770927    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-gq2vz"]
Sep 10 23:36:03.771564 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:03.771177    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz" podUID="0e2ec996-a933-49ba-8620-fb063c377f28" containerName="manager" containerID="cri-o://9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77" gracePeriod=30
Sep 10 23:36:04.017185 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.017163    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:36:04.041932 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.041859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-webhook-cert\") pod \"0e2ec996-a933-49ba-8620-fb063c377f28\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") "
Sep 10 23:36:04.042108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.041929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-metrics-certs\") pod \"0e2ec996-a933-49ba-8620-fb063c377f28\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") "
Sep 10 23:36:04.042108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.041985    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d6nw\" (UniqueName: \"kubernetes.io/projected/0e2ec996-a933-49ba-8620-fb063c377f28-kube-api-access-6d6nw\") pod \"0e2ec996-a933-49ba-8620-fb063c377f28\" (UID: \"0e2ec996-a933-49ba-8620-fb063c377f28\") "
Sep 10 23:36:04.044467 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.044433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0e2ec996-a933-49ba-8620-fb063c377f28" (UID: "0e2ec996-a933-49ba-8620-fb063c377f28"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:36:04.044582 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.044502    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e2ec996-a933-49ba-8620-fb063c377f28" (UID: "0e2ec996-a933-49ba-8620-fb063c377f28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:36:04.044844 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.044819    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2ec996-a933-49ba-8620-fb063c377f28-kube-api-access-6d6nw" (OuterVolumeSpecName: "kube-api-access-6d6nw") pod "0e2ec996-a933-49ba-8620-fb063c377f28" (UID: "0e2ec996-a933-49ba-8620-fb063c377f28"). InnerVolumeSpecName "kube-api-access-6d6nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:36:04.143565 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.143533    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-webhook-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:36:04.143565 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.143560    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e2ec996-a933-49ba-8620-fb063c377f28-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:36:04.143774 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.143573    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6nw\" (UniqueName: \"kubernetes.io/projected/0e2ec996-a933-49ba-8620-fb063c377f28-kube-api-access-6d6nw\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:36:04.255861 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.255830    2564 generic.go:356] "Generic (PLEG): container finished" podID="0e2ec996-a933-49ba-8620-fb063c377f28" containerID="9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77" exitCode=0
Sep 10 23:36:04.255967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.255870    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz"
Sep 10 23:36:04.255967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.255874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz" event={"ID":"0e2ec996-a933-49ba-8620-fb063c377f28","Type":"ContainerDied","Data":"9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77"}
Sep 10 23:36:04.255967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.255910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff55dfbf9-gq2vz" event={"ID":"0e2ec996-a933-49ba-8620-fb063c377f28","Type":"ContainerDied","Data":"9c7121bb9eba31898c9e8e4847d09b503707b12dedc18001c37d364d2fb33472"}
Sep 10 23:36:04.255967 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.255936    2564 scope.go:117] "RemoveContainer" containerID="9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77"
Sep 10 23:36:04.264864 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.264852    2564 scope.go:117] "RemoveContainer" containerID="9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77"
Sep 10 23:36:04.265117 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:36:04.265101    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77\": container with ID starting with 9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77 not found: ID does not exist" containerID="9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77"
Sep 10 23:36:04.265180 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.265125    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77"} err="failed to get container status \"9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77\": rpc error: code = NotFound desc = could not find container \"9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77\": container with ID starting with 9d43ac7f464889a86b31043811c7b09b1e36878913d43296de483108aad32c77 not found: ID does not exist"
Sep 10 23:36:04.271400 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.271372    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-gq2vz"]
Sep 10 23:36:04.273030 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:04.273011    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ff55dfbf9-gq2vz"]
Sep 10 23:36:06.198619 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:06.198583    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2ec996-a933-49ba-8620-fb063c377f28" path="/var/lib/kubelet/pods/0e2ec996-a933-49ba-8620-fb063c377f28/volumes"
Sep 10 23:36:36.270811 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:36.270777    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:36:36.293259 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:36.293237    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:36:36.420649 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:36:36.420623    2564 scope.go:117] "RemoveContainer" containerID="4f589d823071ef1f364cf9fd0679ffa7069e8351d043634cfa3b29bef4c50000"
Sep 10 23:41:29.394093 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"]
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394403    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6a383-7fff-4218-9aaa-405d4b1d4a23" containerName="payload-pre-processing"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394413    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6a383-7fff-4218-9aaa-405d4b1d4a23" containerName="payload-pre-processing"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394422    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c6d351-26e7-4fa6-a977-ddb594c193bd" containerName="payload-processing"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394428    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c6d351-26e7-4fa6-a977-ddb594c193bd" containerName="payload-processing"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394438    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02d0e6-0395-446e-85b0-31b307bc7138" containerName="maas-api"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394444    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02d0e6-0395-446e-85b0-31b307bc7138" containerName="maas-api"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394454    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2ec996-a933-49ba-8620-fb063c377f28" containerName="manager"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394460    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2ec996-a933-49ba-8620-fb063c377f28" containerName="manager"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394468    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36642c88-849b-49eb-b122-3886e0e88428" containerName="maas-api"
Sep 10 23:41:29.394496 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394473    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="36642c88-849b-49eb-b122-3886e0e88428" containerName="maas-api"
Sep 10 23:41:29.394819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394533    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2ec996-a933-49ba-8620-fb063c377f28" containerName="manager"
Sep 10 23:41:29.394819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394543    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02d0e6-0395-446e-85b0-31b307bc7138" containerName="maas-api"
Sep 10 23:41:29.394819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394549    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="36642c88-849b-49eb-b122-3886e0e88428" containerName="maas-api"
Sep 10 23:41:29.394819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394556    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c6d351-26e7-4fa6-a977-ddb594c193bd" containerName="payload-processing"
Sep 10 23:41:29.394819 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.394562    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6a383-7fff-4218-9aaa-405d4b1d4a23" containerName="payload-pre-processing"
Sep 10 23:41:29.397454 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.397435    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.406943 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.406921    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"]
Sep 10 23:41:29.507342 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.507314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-metrics-certs\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.507479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.507352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-maas-api-tls\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.507479 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.507384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-openshift-service-ca\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.507588 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.507493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gjrr\" (UniqueName: \"kubernetes.io/projected/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-kube-api-access-4gjrr\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.608101 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.608074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjrr\" (UniqueName: \"kubernetes.io/projected/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-kube-api-access-4gjrr\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.608221 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.608132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-metrics-certs\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.608221 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.608158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-maas-api-tls\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.608221 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.608186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-openshift-service-ca\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.608867 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.608847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-openshift-service-ca\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.610550 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.610527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-maas-api-tls\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.610647 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.610566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-metrics-certs\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.615859 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.615829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjrr\" (UniqueName: \"kubernetes.io/projected/f8fdd79e-f7ef-4770-859c-2ddf0347ba0a-kube-api-access-4gjrr\") pod \"maas-api-9c7b54f65-7qjmn\" (UID: \"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a\") " pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.709361 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.709331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:29.979968 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.979945    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"]
Sep 10 23:41:29.982187 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:41:29.982158    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fdd79e_f7ef_4770_859c_2ddf0347ba0a.slice/crio-5fe4e91e44d9faa92c04316367ec8341671f313a98f3a1d29ace965779b6e99c WatchSource:0}: Error finding container 5fe4e91e44d9faa92c04316367ec8341671f313a98f3a1d29ace965779b6e99c: Status 404 returned error can't find the container with id 5fe4e91e44d9faa92c04316367ec8341671f313a98f3a1d29ace965779b6e99c
Sep 10 23:41:29.983569 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:29.983552    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:41:30.794756 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:30.794692    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn" event={"ID":"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a","Type":"ContainerStarted","Data":"c8296905811ff2ae1f899977ac9011513fc0cb47f06c0e96367018fce31d7f8a"}
Sep 10 23:41:30.795198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:30.794768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn" event={"ID":"f8fdd79e-f7ef-4770-859c-2ddf0347ba0a","Type":"ContainerStarted","Data":"5fe4e91e44d9faa92c04316367ec8341671f313a98f3a1d29ace965779b6e99c"}
Sep 10 23:41:30.795198 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:30.794879    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:30.812070 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:30.812020    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn" podStartSLOduration=1.522947804 podStartE2EDuration="1.812007515s" totalImagesPullingTime="289.059711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:30.810447484 +0000 UTC m=+2395.096468686" watchObservedRunningTime="2026-09-10 23:41:30.812007515 +0000 UTC m=+2395.098028710"
Sep 10 23:41:36.304888 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:36.304860    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:41:36.324830 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:36.324807    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:41:36.804607 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:36.804582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-9c7b54f65-7qjmn"
Sep 10 23:41:36.853249 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:36.853213    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-htw9m"]
Sep 10 23:41:36.853450 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:36.853415    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m" podUID="d6339823-f27f-4e55-9c19-26ab7a5bb68b" containerName="maas-api" containerID="cri-o://a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710" gracePeriod=30
Sep 10 23:41:38.103288 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.103263    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:41:38.179445 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.179411    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-maas-api-tls\") pod \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") "
Sep 10 23:41:38.179627 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.179468    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-metrics-certs\") pod \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") "
Sep 10 23:41:38.179627 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.179505    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6339823-f27f-4e55-9c19-26ab7a5bb68b-openshift-service-ca\") pod \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") "
Sep 10 23:41:38.179627 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.179571    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/d6339823-f27f-4e55-9c19-26ab7a5bb68b-kube-api-access-ggxcl\") pod \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\" (UID: \"d6339823-f27f-4e55-9c19-26ab7a5bb68b\") "
Sep 10 23:41:38.179993 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.179957    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6339823-f27f-4e55-9c19-26ab7a5bb68b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6339823-f27f-4e55-9c19-26ab7a5bb68b" (UID: "d6339823-f27f-4e55-9c19-26ab7a5bb68b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:41:38.181641 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.181580    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6339823-f27f-4e55-9c19-26ab7a5bb68b" (UID: "d6339823-f27f-4e55-9c19-26ab7a5bb68b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:41:38.181641 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.181597    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6339823-f27f-4e55-9c19-26ab7a5bb68b" (UID: "d6339823-f27f-4e55-9c19-26ab7a5bb68b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:41:38.181859 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.181842    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6339823-f27f-4e55-9c19-26ab7a5bb68b-kube-api-access-ggxcl" (OuterVolumeSpecName: "kube-api-access-ggxcl") pod "d6339823-f27f-4e55-9c19-26ab7a5bb68b" (UID: "d6339823-f27f-4e55-9c19-26ab7a5bb68b"). InnerVolumeSpecName "kube-api-access-ggxcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:41:38.281121 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.281095    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-maas-api-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:41:38.281121 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.281117    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6339823-f27f-4e55-9c19-26ab7a5bb68b-metrics-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:41:38.281121 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.281126    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6339823-f27f-4e55-9c19-26ab7a5bb68b-openshift-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:41:38.281298 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.281134    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/d6339823-f27f-4e55-9c19-26ab7a5bb68b-kube-api-access-ggxcl\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Sep 10 23:41:38.843893 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.843855    2564 generic.go:356] "Generic (PLEG): container finished" podID="d6339823-f27f-4e55-9c19-26ab7a5bb68b" containerID="a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710" exitCode=0
Sep 10 23:41:38.844112 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.843905    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m"
Sep 10 23:41:38.844112 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.843931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m" event={"ID":"d6339823-f27f-4e55-9c19-26ab7a5bb68b","Type":"ContainerDied","Data":"a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710"}
Sep 10 23:41:38.844112 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.843985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777777df97-htw9m" event={"ID":"d6339823-f27f-4e55-9c19-26ab7a5bb68b","Type":"ContainerDied","Data":"c303dd0902faa9193f70e6a23b82c0080dd19dbbb65f17dffa8f3093a2842554"}
Sep 10 23:41:38.844112 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.844009    2564 scope.go:117] "RemoveContainer" containerID="a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710"
Sep 10 23:41:38.853287 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.853264    2564 scope.go:117] "RemoveContainer" containerID="a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710"
Sep 10 23:41:38.853561 ip-10-0-129-219 kubenswrapper[2564]: E0910 23:41:38.853541    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710\": container with ID starting with a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710 not found: ID does not exist" containerID="a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710"
Sep 10 23:41:38.853636 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.853569    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710"} err="failed to get container status \"a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710\": rpc error: code = NotFound desc = could not find container \"a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710\": container with ID starting with a1c396292f7c3b05fc6790eae7eff85edf4ecbf3bede93d6bfae84c20de68710 not found: ID does not exist"
Sep 10 23:41:38.860938 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.860915    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-htw9m"]
Sep 10 23:41:38.864036 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:38.864014    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-777777df97-htw9m"]
Sep 10 23:41:40.197661 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:40.197628    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6339823-f27f-4e55-9c19-26ab7a5bb68b" path="/var/lib/kubelet/pods/d6339823-f27f-4e55-9c19-26ab7a5bb68b/volumes"
Sep 10 23:41:57.437454 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:57.437424    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-d4qt8_3ac2848c-abed-4957-a0f7-80b53ca22e77/manager/0.log"
Sep 10 23:41:57.660740 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:57.660693    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qpfwp_031f92c5-065e-48cf-8919-5064c114610d/server/0.log"
Sep 10 23:41:57.774696 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:57.774629    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rjxmt_95903eff-18ad-4299-bf87-4dc5f442e7b0/manager/0.log"
Sep 10 23:41:58.129129 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:58.129028    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-tq5mf_660c71df-37a3-4c6f-9de5-2a3d74adec72/manager/0.log"
Sep 10 23:41:58.134863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:58.134842    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-tq5mf_660c71df-37a3-4c6f-9de5-2a3d74adec72/copy-manifests/0.log"
Sep 10 23:41:58.463272 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:58.463245    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-9c7b54f65-7qjmn_f8fdd79e-f7ef-4770-859c-2ddf0347ba0a/maas-api/0.log"
Sep 10 23:41:58.892130 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:58.892013    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rd8nk_a84e8cf6-23f7-4e7c-87d6-26dce1e64a40/postgres/0.log"
Sep 10 23:41:59.734028 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:59.733995    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4mqqp_bf51295c-0259-4d53-95f3-2d856081fbe8/manager/0.log"
Sep 10 23:41:59.847677 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:41:59.847647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jfmtk_854df844-debd-457d-911c-672e87cdb96c/manager/0.log"
Sep 10 23:42:00.757732 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:00.757682    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dh2sn_380c33da-7499-41ca-8b55-b4251aa997ba/discovery/0.log"
Sep 10 23:42:00.981764 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:00.981733    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-xltjl_ddd56ef4-0e41-4343-ae31-3e7c4f565591/kube-auth-proxy/0.log"
Sep 10 23:42:01.205682 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:01.205655    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bdnnx_0244d17f-d128-47dd-a306-7593b8f2191e/payload-pre-processing/0.log"
Sep 10 23:42:01.869372 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:01.869343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2pcjv_38e1d1a4-c40f-4254-bc70-8463956df8bd/main/0.log"
Sep 10 23:42:01.975856 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:01.975825    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vrfcj_5aaaca42-9bdf-4d23-85e2-444b921ea26c/main/0.log"
Sep 10 23:42:02.086097 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:02.086072    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zssvn_626fd44f-d0c1-4246-b6a5-011169537b37/main/0.log"
Sep 10 23:42:02.192767 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:02.192744    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fx75qd_e9c08499-f7a6-4d03-9092-7a6d1c79ee94/main/0.log"
Sep 10 23:42:02.297346 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:02.297325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-c6rgj_8a18e2e9-8858-45b0-9f54-264b4912eaa0/main/0.log"
Sep 10 23:42:10.859060 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:10.859027    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-psv9l_9b53e2c5-97cd-4de1-a418-40e5fe116a6f/global-pull-secret-syncer/0.log"
Sep 10 23:42:11.011056 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:11.011023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hwkpx_5791960f-3106-459f-bfad-3ca157fbeec3/konnectivity-agent/0.log"
Sep 10 23:42:11.059378 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:11.059346    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-219.ec2.internal_586716cd581743efb727a55187e31d8e/haproxy/0.log"
Sep 10 23:42:16.099600 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:16.099567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4mqqp_bf51295c-0259-4d53-95f3-2d856081fbe8/manager/0.log"
Sep 10 23:42:16.128316 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:16.128289    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jfmtk_854df844-debd-457d-911c-672e87cdb96c/manager/0.log"
Sep 10 23:42:18.491969 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:18.491945    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slfmv_d7672bee-4336-4eb2-9369-846e0c36db51/node-exporter/0.log"
Sep 10 23:42:18.514051 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:18.514022    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slfmv_d7672bee-4336-4eb2-9369-846e0c36db51/kube-rbac-proxy/0.log"
Sep 10 23:42:18.541986 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:18.541933    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slfmv_d7672bee-4336-4eb2-9369-846e0c36db51/init-textfile/0.log"
Sep 10 23:42:21.231369 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.231334    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/1.log"
Sep 10 23:42:21.235924 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.235905    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9p9ht_54d614b8-727f-4d5f-8fd0-77390fe3200a/console-operator/2.log"
Sep 10 23:42:21.764678 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.764642    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"]
Sep 10 23:42:21.765040 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.765026    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6339823-f27f-4e55-9c19-26ab7a5bb68b" containerName="maas-api"
Sep 10 23:42:21.765086 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.765041    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6339823-f27f-4e55-9c19-26ab7a5bb68b" containerName="maas-api"
Sep 10 23:42:21.765131 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.765109    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6339823-f27f-4e55-9c19-26ab7a5bb68b" containerName="maas-api"
Sep 10 23:42:21.768202 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.768182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.770412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.770390    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xqw8\"/\"openshift-service-ca.crt\""
Sep 10 23:42:21.770504 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.770413    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5xqw8\"/\"default-dockercfg-8m8wk\""
Sep 10 23:42:21.771108 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.771094    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xqw8\"/\"kube-root-ca.crt\""
Sep 10 23:42:21.777670 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.777649    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"]
Sep 10 23:42:21.819778 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.819758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-podres\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.819888 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.819790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-lib-modules\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.819888 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.819841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstt7\" (UniqueName: \"kubernetes.io/projected/af6f0053-92dc-4065-8018-f198d61a2066-kube-api-access-jstt7\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.820012 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.819957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-sys\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.820012 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.819987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-proc\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921436 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstt7\" (UniqueName: \"kubernetes.io/projected/af6f0053-92dc-4065-8018-f198d61a2066-kube-api-access-jstt7\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921539 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-sys\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921600 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921538    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-proc\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921600 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-podres\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921746 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-lib-modules\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921746 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-sys\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921746 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-proc\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-podres\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.921863 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.921798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6f0053-92dc-4065-8018-f198d61a2066-lib-modules\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:21.928730 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:21.928680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstt7\" (UniqueName: \"kubernetes.io/projected/af6f0053-92dc-4065-8018-f198d61a2066-kube-api-access-jstt7\") pod \"perf-node-gather-daemonset-n5grm\" (UID: \"af6f0053-92dc-4065-8018-f198d61a2066\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:22.078372 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:22.078307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:22.347804 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:22.347715    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"]
Sep 10 23:42:22.350361 ip-10-0-129-219 kubenswrapper[2564]: W0910 23:42:22.350336    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaf6f0053_92dc_4065_8018_f198d61a2066.slice/crio-52bb26fa4e631ea2e43fbd7b5da732711acc13ab211cb2a77aaa2eae0afb904b WatchSource:0}: Error finding container 52bb26fa4e631ea2e43fbd7b5da732711acc13ab211cb2a77aaa2eae0afb904b: Status 404 returned error can't find the container with id 52bb26fa4e631ea2e43fbd7b5da732711acc13ab211cb2a77aaa2eae0afb904b
Sep 10 23:42:22.549992 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:22.549967    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-8drn6_a1de3337-6e3d-4da9-b4a5-6ccf3616bcff/volume-data-source-validator/0.log"
Sep 10 23:42:23.059965 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.059926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm" event={"ID":"af6f0053-92dc-4065-8018-f198d61a2066","Type":"ContainerStarted","Data":"ee6638044ed17b39743c0ab49706a1beb94581a3754c69c7343658c83e92c7bb"}
Sep 10 23:42:23.060130 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.059981    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm" event={"ID":"af6f0053-92dc-4065-8018-f198d61a2066","Type":"ContainerStarted","Data":"52bb26fa4e631ea2e43fbd7b5da732711acc13ab211cb2a77aaa2eae0afb904b"}
Sep 10 23:42:23.060130 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.060066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:23.075833 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.075793    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm" podStartSLOduration=2.07577865 podStartE2EDuration="2.07577865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:23.074840204 +0000 UTC m=+2447.360861405" watchObservedRunningTime="2026-09-10 23:42:23.07577865 +0000 UTC m=+2447.361799851"
Sep 10 23:42:23.639827 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.639801    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8szzn_b9bed0bc-550e-4da3-9a09-223ff72d236e/dns/0.log"
Sep 10 23:42:23.657892 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.657868    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8szzn_b9bed0bc-550e-4da3-9a09-223ff72d236e/kube-rbac-proxy/0.log"
Sep 10 23:42:23.808555 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:23.808532    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-grwkz_a17e0c8e-f203-4b9f-a43d-8c5e2f8f64da/dns-node-resolver/0.log"
Sep 10 23:42:24.353842 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:24.353814    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9k8bl_4ac069be-2664-461a-8d5f-0cb231d3b4d9/node-ca/0.log"
Sep 10 23:42:25.584151 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:25.584120    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dh2sn_380c33da-7499-41ca-8b55-b4251aa997ba/discovery/0.log"
Sep 10 23:42:25.634762 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:25.634738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-xltjl_ddd56ef4-0e41-4343-ae31-3e7c4f565591/kube-auth-proxy/0.log"
Sep 10 23:42:25.695901 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:25.695880    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bdnnx_0244d17f-d128-47dd-a306-7593b8f2191e/payload-pre-processing/0.log"
Sep 10 23:42:26.683370 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:26.683338    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srh7k_e99ce852-5695-4da2-a352-5d714874db49/serve-healthcheck-canary/0.log"
Sep 10 23:42:27.189084 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:27.189036    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6685x_512332f9-1be2-462f-87db-a27b8c499f33/kube-rbac-proxy/0.log"
Sep 10 23:42:27.207639 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:27.207621    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6685x_512332f9-1be2-462f-87db-a27b8c499f33/exporter/0.log"
Sep 10 23:42:27.227375 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:27.227353    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6685x_512332f9-1be2-462f-87db-a27b8c499f33/extractor/0.log"
Sep 10 23:42:29.074612 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:29.074580    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-n5grm"
Sep 10 23:42:29.647279 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:29.647235    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-d4qt8_3ac2848c-abed-4957-a0f7-80b53ca22e77/manager/0.log"
Sep 10 23:42:29.712849 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:29.712820    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qpfwp_031f92c5-065e-48cf-8919-5064c114610d/server/0.log"
Sep 10 23:42:29.790792 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:29.790761    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rjxmt_95903eff-18ad-4299-bf87-4dc5f442e7b0/manager/0.log"
Sep 10 23:42:29.942412 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:29.942387    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-tq5mf_660c71df-37a3-4c6f-9de5-2a3d74adec72/manager/0.log"
Sep 10 23:42:29.960066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:29.960041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-tq5mf_660c71df-37a3-4c6f-9de5-2a3d74adec72/copy-manifests/0.log"
Sep 10 23:42:32.608066 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:32.608024    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dh2sn_380c33da-7499-41ca-8b55-b4251aa997ba/discovery/0.log"
Sep 10 23:42:32.634782 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:32.634760    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-xltjl_ddd56ef4-0e41-4343-ae31-3e7c4f565591/kube-auth-proxy/0.log"
Sep 10 23:42:32.693907 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:32.693884    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bdnnx_0244d17f-d128-47dd-a306-7593b8f2191e/payload-pre-processing/0.log"
Sep 10 23:42:34.614573 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:34.614543    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dh2sn_380c33da-7499-41ca-8b55-b4251aa997ba/discovery/0.log"
Sep 10 23:42:34.647451 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:34.647429    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-xltjl_ddd56ef4-0e41-4343-ae31-3e7c4f565591/kube-auth-proxy/0.log"
Sep 10 23:42:34.691173 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:34.691144    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bdnnx_0244d17f-d128-47dd-a306-7593b8f2191e/payload-pre-processing/0.log"
Sep 10 23:42:36.960719 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:36.960678    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5l8bb_01171f17-339c-4d81-a52f-210d94357f06/kube-storage-version-migrator-operator/1.log"
Sep 10 23:42:36.961871 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:36.961843    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5l8bb_01171f17-339c-4d81-a52f-210d94357f06/kube-storage-version-migrator-operator/0.log"
Sep 10 23:42:37.918075 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:37.918000    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-98f8k_6c049943-25e6-47aa-b615-1ac1c5707d84/kube-multus/0.log"
Sep 10 23:42:38.228034 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.228003    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/kube-multus-additional-cni-plugins/0.log"
Sep 10 23:42:38.246231 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.246209    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/egress-router-binary-copy/0.log"
Sep 10 23:42:38.268030 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.268006    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/cni-plugins/0.log"
Sep 10 23:42:38.288051 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.288027    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/bond-cni-plugin/0.log"
Sep 10 23:42:38.307099 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.307076    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/routeoverride-cni/0.log"
Sep 10 23:42:38.325373 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.325353    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/whereabouts-cni-bincopy/0.log"
Sep 10 23:42:38.344147 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.344128    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xhqtb_347d9a42-6aa2-4615-b7fd-0ae132d0f512/whereabouts-cni/0.log"
Sep 10 23:42:38.463308 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.463286    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9vlkd_6ae183f5-9da6-46d2-b227-78d5de9e33a5/network-metrics-daemon/0.log"
Sep 10 23:42:38.479850 ip-10-0-129-219 kubenswrapper[2564]: I0910 23:42:38.479798    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9vlkd_6ae183f5-9da6-46d2-b227-78d5de9e33a5/kube-rbac-proxy/0.log"