Sep 11 12:14:24.483737 ip-10-0-135-5 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 12:14:24.914715 ip-10-0-135-5 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:14:24.914715 ip-10-0-135-5 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 12:14:24.914715 ip-10-0-135-5 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:14:24.914715 ip-10-0-135-5 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 12:14:24.914715 ip-10-0-135-5 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:14:24.916767 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.916664    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 12:14:24.921609 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921580    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:24.921609 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921604    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:24.921609 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921608    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:24.921609 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921611    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:24.921609 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921615    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:24.921609 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921618    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921621    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921625    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921628    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921631    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921634    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921636    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921639    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921643    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921646    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921649    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921651    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921654    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921657    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921659    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921662    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921666    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921670    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921673    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:24.921863 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921676    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921679    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921682    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921685    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921689    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921699    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921702    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921705    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921708    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921711    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921713    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921716    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921718    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921721    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921723    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921727    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921730    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921732    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921735    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921738    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:24.922318 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921740    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921743    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921746    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921748    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921751    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921753    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921756    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921758    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921761    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921771    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921775    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921779    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921781    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921784    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921786    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921789    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921792    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921795    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921798    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921800    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:24.922840 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921803    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921806    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921809    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921812    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921815    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921817    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921820    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921824    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921827    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921830    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921833    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921836    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921839    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921841    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921844    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921848    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921851    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921854    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921856    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921859    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:24.923327 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921862    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921865    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921868    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921872    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921874    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921877    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921880    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921883    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921887    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921890    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921893    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921896    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921899    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921901    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.921904    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922659    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922667    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922671    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922674    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:24.923848 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922679    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922683    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922686    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922689    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922692    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922695    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922698    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922701    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922704    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922707    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922710    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922712    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922715    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922719    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922721    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922724    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922726    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922729    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922732    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922735    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:24.924304 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922737    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922740    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922742    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922745    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922748    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922750    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922753    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922755    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922758    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922760    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922762    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922765    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922768    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922773    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922776    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922780    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922782    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922786    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922789    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:24.924832 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922791    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922794    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922797    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922800    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922802    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922805    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922807    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922810    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922813    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922816    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922818    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922821    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922824    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922826    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922829    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922832    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922834    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922838    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922841    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:24.925303 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922845    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922848    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922851    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922854    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922856    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922859    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922863    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922866    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922869    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922872    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922875    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922877    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922880    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922882    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922886    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922888    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922891    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922895    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922899    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922902    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:24.925801 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922905    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922908    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922911    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922914    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922916    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922919    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922922    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922924    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922927    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922929    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922932    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922935    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922937    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922940    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922942    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922945    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.922947    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923340    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923349    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923352    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:24.926324 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923355    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923358    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923361    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923363    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923366    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923369    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923371    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923374    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923377    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923379    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923382    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923384    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923387    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923390    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923408    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923413    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923416    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923420    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923423    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923425    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:24.926847 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923428    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923430    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923433    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923436    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923438    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923441    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923443    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923446    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923448    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923451    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923455    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923458    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923461    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923464    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923466    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923469    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923472    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923474    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923477    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:24.927367 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923481    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923485    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923488    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923491    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923494    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923497    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923500    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923503    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923506    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923509    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923513    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923515    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923518    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923520    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923523    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923525    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923528    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923530    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923533    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923535    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:24.927900 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923538    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923541    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923543    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923548    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923551    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923553    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923556    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923559    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923561    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923564    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923566    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923569    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923571    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923575    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923579    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923581    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923584    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923587    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923591    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:24.928431 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923594    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923597    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923600    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923603    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923605    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923608    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923610    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923613    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923615    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923618    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923621    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923623    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923626    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923628    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923631    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923634    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923639    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.923641    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923722    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923764    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923771    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 12:14:24.928901 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923775    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923779    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923783    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923787    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923792    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923795    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923798    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923802    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923805    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923808    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923812    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923815    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923818    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923822    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923825    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923830    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923833    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923836    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923840    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923843    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923847    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923850    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923853    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 12:14:24.929488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923857    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923860    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923863    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923867    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923872    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923875    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923879    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923883    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923886    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923889    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923892    2561 flags.go:64] FLAG: --enable-server="true"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923895    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923899    2561 flags.go:64] FLAG: --event-burst="100"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923902    2561 flags.go:64] FLAG: --event-qps="50"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923905    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923908    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923911    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923915    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923918    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923921    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923924    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923927    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923930    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923934    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923937    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 12:14:24.930062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923940    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923943    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923946    2561 flags.go:64] FLAG: --feature-gates=""
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923950    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923954    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923957    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923960    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923963    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923967    2561 flags.go:64] FLAG: --help="false"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923970    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-135-5.ec2.internal"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923973    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923977    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923981    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923984    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923987    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923991    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923994    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.923997    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924000    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924003    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924006    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924009    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924013    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924016    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 12:14:24.930699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924019    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924022    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924025    2561 flags.go:64] FLAG: --lock-file=""
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924028    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924031    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924034    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924039    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924043    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924046    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924049    2561 flags.go:64] FLAG: --logging-format="text"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924052    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924055    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924058    2561 flags.go:64] FLAG: --manifest-url=""
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924061    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924065    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924068    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924072    2561 flags.go:64] FLAG: --max-pods="110"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924076    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924079    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924085    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924088    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924091    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924095    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924098    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924106    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 12:14:24.931358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924109    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924112    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924115    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924118    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924125    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924128    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924131    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924134    2561 flags.go:64] FLAG: --port="10250"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924137    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924140    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057a8f8c33b239ecf"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924144    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924147    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924150    2561 flags.go:64] FLAG: --register-node="true"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924153    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924157    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924160    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924163    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924166    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924169    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924173    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924176    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924179    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924182    2561 flags.go:64] FLAG: --runonce="false"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924185    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924188    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 12:14:24.932028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924191    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924194    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924199    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924202    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924205    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924208    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924211    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924214    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924217    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924220    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924223    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924226    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924229    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924235    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924237    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924240    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924245    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924248    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924251    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924254    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924257    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924260    2561 flags.go:64] FLAG: --v="2"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924264    2561 flags.go:64] FLAG: --version="false"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924269    2561 flags.go:64] FLAG: --vmodule=""
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924272    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 12:14:24.932716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.924276    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924409    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924414    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924417    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924420    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924423    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924425    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924429    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924433    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924437    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924440    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924443    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924445    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924448    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924451    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924454    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924456    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924459    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924462    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924465    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:24.933543 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924467    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924470    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924473    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924477    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924481    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924484    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924487    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924491    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924496    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924499    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924503    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924508    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924511    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924514    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924517    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924520    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924523    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924525    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924528    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:24.934069 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924531    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924535    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924539    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924542    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924545    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924548    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924550    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924553    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924556    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924558    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924561    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924564    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924567    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924570    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924572    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924575    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924577    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924580    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924583    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924585    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:24.934576 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924588    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924591    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924593    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924596    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924598    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924602    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924605    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924607    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924610    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924613    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924616    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924618    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924621    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924625    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924629    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924632    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924634    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924637    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924639    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924642    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:24.935072 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924645    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924647    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924651    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924656    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924658    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924661    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924664    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924666    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924669    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924671    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924674    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924677    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924680    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924683    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924685    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924689    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924693    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924696    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924699    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:24.935593 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924702    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.924705    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.925542    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.932881    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.932905    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.933641    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.933649    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933716    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933724    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933727    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933731    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933733    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933736    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933740    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:24.936078 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933742    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933747    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933751    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933754    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933757    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933761    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933764    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933775    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933778    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933781    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933783    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933786    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933789    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933793    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933797    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933800    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933803    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933805    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933808    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:24.936500 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933812    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933815    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933818    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933831    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933834    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933837    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933840    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933843    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933846    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933848    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933851    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933854    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933856    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933859    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933862    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933864    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933868    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933871    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933873    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933876    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:24.936977 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933879    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933882    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933884    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933887    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933889    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933892    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933895    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933898    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933900    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933903    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933905    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933908    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933911    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933914    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933916    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933919    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933923    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933926    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933928    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933931    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:24.937482 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933934    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933937    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933939    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933942    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933944    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933947    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933950    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933953    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933955    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933960    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933965    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933967    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933971    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933974    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933976    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933979    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933982    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933984    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933987    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:24.938002 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933990    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933992    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933995    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.933997    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934000    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934003    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934005    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934009    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934011    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934014    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934017    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934020    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934023    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934025    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.934031    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:14:24.938531 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934177    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934185    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934189    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934192    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934195    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934198    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934200    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934203    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934206    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934209    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934212    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934215    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934218    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934220    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934223    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934226    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934229    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934231    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934235    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:24.938915 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934238    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934242    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934244    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934247    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934250    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934252    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934255    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934258    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934261    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934266    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934269    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934272    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934275    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934278    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934281    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934283    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934286    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934289    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934291    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:24.939392 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934294    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934297    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934300    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934304    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934307    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934310    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934312    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934315    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934318    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934320    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934323    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934325    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934328    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934331    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934333    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934336    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934338    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934341    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934343    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934346    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:24.939962 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934349    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934352    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934354    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934358    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934360    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934363    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934365    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934368    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934370    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934373    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934375    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934378    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934380    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934383    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934385    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934388    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934391    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934394    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934411    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934413    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:24.940478 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934416    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934419    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934422    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934425    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934427    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934430    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934433    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934436    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934438    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934441    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934443    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934446    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934449    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934452    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934455    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934458    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934461    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934464    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934467    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934469    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:24.940960 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:24.934472    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:24.941472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.934477    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:14:24.941472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.934619    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 12:14:24.941472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.937877    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 12:14:24.941472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.938894    2561 server.go:1019] "Starting client certificate rotation"
Sep 11 12:14:24.941472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.939002    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 12:14:24.941472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.939036    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 12:14:24.942153 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.942131    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 12:14:24.948330 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.948299    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 12:14:24.958834 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.958808    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 11 12:14:24.962058 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.962036    2561 log.go:25] "Validated CRI v1 image API"
Sep 11 12:14:24.962875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.962860    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 12:14:24.968055 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.968030    2561 fs.go:135] Filesystem UUIDs: map[18b4c3b6-2b3c-4519-8c88-22a7592d3533:/dev/nvme0n1p3 567846d0-eb55-449b-b642-84b1eb0614ad:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 11 12:14:24.968142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.968052    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 12:14:24.972434 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.972308    2561 manager.go:216] Machine: {Timestamp:2026-09-11 12:14:24.971828723 +0000 UTC m=+0.362366056 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103226 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f0a452b30788297886d41749e03b SystemUUID:ec27f0a4-52b3-0788-2978-86d41749e03b BootID:7da1c95c-7fe3-4b44-9510-fc9799843be1 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:7b:c7:bc:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:7b:c7:bc:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:68:29:f9:08:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 12:14:24.972434 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.972429    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 12:14:24.972531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.972518    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 12:14:24.975571 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.975537    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 12:14:24.975715 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.975573    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 12:14:24.976914 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.976902    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 12:14:24.976955 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.976917    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 12:14:24.976955 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.976926    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 12:14:24.978065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.978053    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 12:14:24.978836 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.978825    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 12:14:24.982246 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982229    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 12:14:24.982389 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982379    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 12:14:24.982443 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982427    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 12:14:24.982484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982448    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 12:14:24.982484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982464    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 12:14:24.982484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982474    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 11 12:14:24.982607 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982486    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 12:14:24.982934 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982922    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 12:14:24.982968 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.982943    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 12:14:24.985508 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.985485    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 12:14:24.988011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.987995    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 12:14:24.988091 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.988027    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 12:14:24.989545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989528    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989549    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989558    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989565    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989574    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989580    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989594    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 12:14:24.989609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989603    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 12:14:24.989875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989620    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 12:14:24.989875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989627    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 12:14:24.989875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989638    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 12:14:24.989875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.989645    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 12:14:24.990664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.990646    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 12:14:24.990664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.990667    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 12:14:24.993812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.993758    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 12:14:24.993926 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.993903    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 12:14:24.994918 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:24.994891    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 12:14:24.995026 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:24.995003    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 12:14:24.997965 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.997952    2561 server.go:1268] "Started kubelet"
Sep 11 12:14:24.998069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.998032    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 12:14:24.998163 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.998032    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 12:14:24.998213 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.998163    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 12:14:24.999037 ip-10-0-135-5 systemd[1]: Started Kubernetes Kubelet.
Sep 11 12:14:24.999291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:24.999275    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 12:14:25.001154 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.001140    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 12:14:25.004372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.004357    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 12:14:25.004372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.004366    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 12:14:25.004523 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.004477    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.004948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.004927    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 12:14:25.004948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.004929    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 12:14:25.005075 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.004957    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 12:14:25.005131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005105    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 12:14:25.005131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005115    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 12:14:25.005521 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005502    2561 factory.go:55] Registering systemd factory
Sep 11 12:14:25.005589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005526    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 11 12:14:25.005879 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005865    2561 factory.go:153] Registering CRI-O factory
Sep 11 12:14:25.005948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005882    2561 factory.go:223] Registration of the crio container factory successfully
Sep 11 12:14:25.005948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005937    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 12:14:25.006053 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005958    2561 factory.go:103] Registering Raw factory
Sep 11 12:14:25.006053 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.005974    2561 manager.go:1196] Started watching for new ooms in manager
Sep 11 12:14:25.006203 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.006181    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 12:14:25.006380 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.006367    2561 manager.go:349] Starting recovery of all containers
Sep 11 12:14:25.006455 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.006374    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 12:14:25.007012 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.005997    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d44339379946d5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 12:14:24.997918421 +0000 UTC m=+0.388455754,LastTimestamp:2026-09-11 12:14:24.997918421 +0000 UTC m=+0.388455754,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 12:14:25.016763 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.016737    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mc45w"
Sep 11 12:14:25.016873 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.016779    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 12:14:25.017339 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.017322    2561 manager.go:354] Recovery completed
Sep 11 12:14:25.023966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.023838    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.025972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.025950    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.026048 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.026010    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.026048 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.026021    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.026664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.026643    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 12:14:25.026664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.026662    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 12:14:25.026799 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.026681    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 12:14:25.026799 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.026646    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.027695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.027548    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mc45w"
Sep 11 12:14:25.027888 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.027800    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4433939454544  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 12:14:25.025967428 +0000 UTC m=+0.416504761,LastTimestamp:2026-09-11 12:14:25.025967428 +0000 UTC m=+0.416504761,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 12:14:25.028065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.028047    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.028126 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.028079    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.028126 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.028096    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.028741 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.028724    2561 policy_none.go:49] "None policy: Start"
Sep 11 12:14:25.028804 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.028748    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 12:14:25.028804 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.028763    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 12:14:25.029928 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.029916    2561 policy_none.go:47] "Start"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072148    2561 manager.go:337] "Starting Device Plugin manager"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.072198    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072210    2561 server.go:87] "Starting device plugin registration server"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072680    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072692    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072789    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072878    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.072888    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.073388    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 12:14:25.073844 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.073447    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.098378 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.098332    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 12:14:25.099832 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.099806    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 12:14:25.099956 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.099869    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 12:14:25.099956 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.099899    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 12:14:25.099956 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.099947    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 12:14:25.104302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.104279    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:25.173560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.173457    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.174664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.174645    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.174770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.174677    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.174770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.174691    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.174770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.174719    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.188670 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.188647    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.188767 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.188679    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-5.ec2.internal\": node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.200908 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.200881    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"]
Sep 11 12:14:25.201041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.200960    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.202385 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.202369    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.202483 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.202408    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.202483 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.202423    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.204535 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.204521    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.204635 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.204620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.204693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.204651    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.205418 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205378    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.205418 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205378    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.205522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205429    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.205522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205435    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.205522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205445    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.205522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205449    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.205663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.205663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/243f33b0da41e521566de787222b729f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.ec2.internal\" (UID: \"243f33b0da41e521566de787222b729f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.205663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.205576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.207629 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.207615    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.207700 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.207641    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:25.208433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.208417    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:25.208528 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.208444    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:25.208528 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.208454    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:25.221735 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.221706    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.239314 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.239289    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.243832 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.243813    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.305983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.305948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.305983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.305987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.306200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.306011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/243f33b0da41e521566de787222b729f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.ec2.internal\" (UID: \"243f33b0da41e521566de787222b729f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.306200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.306028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.306200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.306057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/243f33b0da41e521566de787222b729f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.ec2.internal\" (UID: \"243f33b0da41e521566de787222b729f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.306200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.306101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.322665 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.322633    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.423824 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.423722    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.524375 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.524319    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.542910 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.542877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.546145 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.546127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:25.625393 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.625332    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.726418 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.726307    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.827386 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.827350    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.928467 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:25.928425    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:25.938336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.938309    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 12:14:25.938525 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:25.938502    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 12:14:26.029104 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:26.029012    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:26.032338 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.032296    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 12:09:25 +0000 UTC" deadline="2028-03-11 20:27:58.30185816 +0000 UTC"
Sep 11 12:14:26.032338 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.032327    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13136h13m32.26953457s"
Sep 11 12:14:26.129562 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:26.129529    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:26.225241 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:26.225205    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3255f401d21cf30d3c48f303e873f0.slice/crio-98bcbb79873567b54a44655ddebf36ad866f1752718771de520f2f828ff3590e WatchSource:0}: Error finding container 98bcbb79873567b54a44655ddebf36ad866f1752718771de520f2f828ff3590e: Status 404 returned error can't find the container with id 98bcbb79873567b54a44655ddebf36ad866f1752718771de520f2f828ff3590e
Sep 11 12:14:26.229724 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:26.229700    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:26.231011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.230996    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:14:26.260801 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.260769    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:26.330728 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:26.330645    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:26.365935 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:26.365899    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243f33b0da41e521566de787222b729f.slice/crio-56ddc60df6c3b89be3a9e42e5796dc6b7c11caed1c9815e82f936d097d3b3d12 WatchSource:0}: Error finding container 56ddc60df6c3b89be3a9e42e5796dc6b7c11caed1c9815e82f936d097d3b3d12: Status 404 returned error can't find the container with id 56ddc60df6c3b89be3a9e42e5796dc6b7c11caed1c9815e82f936d097d3b3d12
Sep 11 12:14:26.431521 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:26.431483    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 12:14:26.468534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.468507    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:26.474391 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.474362    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:26.505954 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.505916    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:26.519800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.519776    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 12:14:26.520875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.520861    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 12:14:26.539674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.539653    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 12:14:26.986474 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.986435    2561 apiserver.go:52] "Watching apiserver"
Sep 11 12:14:26.995815 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.995784    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 12:14:26.996221 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.996171    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal","openshift-multus/multus-72m7f","openshift-multus/multus-additional-cni-plugins-jlgq7","openshift-multus/network-metrics-daemon-vb9f7","kube-system/global-pull-secret-syncer-2cbzp","openshift-dns/node-resolver-mhljr","openshift-image-registry/node-ca-6b29g","openshift-network-diagnostics/network-check-target-j4zck","openshift-network-operator/iptables-alerter-cbhkm","openshift-ovn-kubernetes/ovnkube-node-5szwd","kube-system/konnectivity-agent-9c6rx","kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm","openshift-cluster-node-tuning-operator/tuned-stnqf"]
Sep 11 12:14:26.999134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:26.999112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.001353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.001329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.003827 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.003801    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.005424 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.005403    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 12:14:27.006153 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.006131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.006256 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.006201    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:27.008431 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.008414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.008509 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.008476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:27.010668 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.010647    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.012941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.012917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.015397 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.015376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.017981 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.017955    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 12:14:27.018090 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.018021    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.018090 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.018041    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.018200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.018176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.018200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.018193    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.018307 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.017957    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w9vbg\""
Sep 11 12:14:27.018307 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.017958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 12:14:27.018703 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.018569    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 12:14:27.018975 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.018957    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs9r5\""
Sep 11 12:14:27.019143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019123    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 12:14:27.019280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019263    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.019347 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cnibin\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019402 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019402 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfh72\" (UniqueName: \"kubernetes.io/projected/e7a698e7-ea0e-4c2b-afba-7c153c612e02-kube-api-access-dfh72\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-registration-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-etc-selinux\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019531    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.019534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019568    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrr9v\" (UniqueName: \"kubernetes.io/projected/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-kube-api-access-jrr9v\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-sys-fs\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019635    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lvqds\""
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckrc\" (UniqueName: \"kubernetes.io/projected/9e2a7c85-b9e5-4879-a00d-d73018cf4252-kube-api-access-vckrc\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-system-cni-dir\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dfc139d-214b-4e2f-8424-f97bc38788a0-host\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h85m7\""
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2dfc139d-214b-4e2f-8424-f97bc38788a0-serviceca\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019877    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pd8gb\""
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5z6\" (UniqueName: \"kubernetes.io/projected/2dfc139d-214b-4e2f-8424-f97bc38788a0-kube-api-access-5v5z6\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.019906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-socket-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.020436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.020436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.019968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-device-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.020436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.020308    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.020436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.020340    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.021005 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.020988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lrgl6\""
Sep 11 12:14:27.021143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.021126    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.021269 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.021254    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.025790 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.025758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.026423 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.026397    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.026635 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.026621    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2r7jt\""
Sep 11 12:14:27.027455 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.027440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 12:14:27.027793 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.027772    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 12:14:27.028250 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.028234    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:27.028331 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.028309    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:27.028556 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.028543    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n6hql\""
Sep 11 12:14:27.031864 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.031840    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 12:14:27.031864 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.031852    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 12:14:27.032007 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.031873    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 12:14:27.032007 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.031900    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.033038 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.033020    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-frznq\""
Sep 11 12:14:27.033347 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.033332    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 12:14:27.033835 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.033808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 12:14:27.034015 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.033999    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 12:14:27.036750 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.036736    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 12:14:27.046530 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.046510    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 12:14:27.086041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.086008    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:27.105337 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.105275    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerStarted","Data":"98bcbb79873567b54a44655ddebf36ad866f1752718771de520f2f828ff3590e"}
Sep 11 12:14:27.106165 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.106144    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 12:14:27.106669 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.106648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal" event={"ID":"243f33b0da41e521566de787222b729f","Type":"ContainerStarted","Data":"56ddc60df6c3b89be3a9e42e5796dc6b7c11caed1c9815e82f936d097d3b3d12"}
Sep 11 12:14:27.120843 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.120984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.120984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-kubelet\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.120984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-modprobe-d\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.120984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysctl-d\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.120984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovnkube-script-lib\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.120984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-run\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.120988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-socket-dir-parent\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmx7\" (UniqueName: \"kubernetes.io/projected/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-kube-api-access-6kmx7\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-systemd\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-var-lib-kubelet\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcttw\" (UniqueName: \"kubernetes.io/projected/36901f08-aeb6-4775-b8f3-a8057c5afe33-kube-api-access-lcttw\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-device-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.121296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cnibin\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cnibin\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-device-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfh72\" (UniqueName: \"kubernetes.io/projected/e7a698e7-ea0e-4c2b-afba-7c153c612e02-kube-api-access-dfh72\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-cni-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-cni-bin\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-multus-certs\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysctl-conf\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-etc-selinux\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-etc-selinux\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-etc-kubernetes\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/041a90da-ab4a-47b2-b4b5-463f9a448d17-tmp\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.121813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-systemd-units\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-slash\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-run-netns\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61b4af8a-0798-441f-84b5-4c24cb5e3839-cni-binary-copy\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-iptables-alerter-script\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.121995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-tuned\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-log-socket\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-system-cni-dir\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-socket-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-system-cni-dir\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63bae1b4-810c-4092-a4ba-cc816e956adc-hosts-file\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-system-cni-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-cni-multus\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.122491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-socket-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xvb\" (UniqueName: \"kubernetes.io/projected/041a90da-ab4a-47b2-b4b5-463f9a448d17-kube-api-access-87xvb\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-kubelet\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-systemd\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-ovn\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-run-ovn-kubernetes\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23350a94-a70f-4322-8fad-a5ddf5152cf4-konnectivity-ca\") pod \"konnectivity-agent-9c6rx\" (UID: \"23350a94-a70f-4322-8fad-a5ddf5152cf4\") " pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87a282b2-6cba-47e6-8831-de744de9a5af-dbus\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bae1b4-810c-4092-a4ba-cc816e956adc-tmp-dir\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-sys-fs\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5ph\" (UniqueName: \"kubernetes.io/projected/63bae1b4-810c-4092-a4ba-cc816e956adc-kube-api-access-vr5ph\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-netns\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-lib-modules\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-var-lib-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-sys-fs\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dfc139d-214b-4e2f-8424-f97bc38788a0-host\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.123244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2dfc139d-214b-4e2f-8424-f97bc38788a0-serviceca\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5z6\" (UniqueName: \"kubernetes.io/projected/2dfc139d-214b-4e2f-8424-f97bc38788a0-kube-api-access-5v5z6\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-hostroot\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dfc139d-214b-4e2f-8424-f97bc38788a0-host\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysconfig\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrr9v\" (UniqueName: \"kubernetes.io/projected/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-kube-api-access-jrr9v\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-cnibin\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-conf-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5nw\" (UniqueName: \"kubernetes.io/projected/61b4af8a-0798-441f-84b5-4c24cb5e3839-kube-api-access-xj5nw\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-host\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.122911    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-etc-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.122955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckrc\" (UniqueName: \"kubernetes.io/projected/9e2a7c85-b9e5-4879-a00d-d73018cf4252-kube-api-access-vckrc\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.122991    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:27.622951654 +0000 UTC m=+3.013488980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87a282b2-6cba-47e6-8831-de744de9a5af-kubelet-config\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-host-slash\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.123784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-kubernetes\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2dfc139d-214b-4e2f-8424-f97bc38788a0-serviceca\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-node-log\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-cni-netd\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovnkube-config\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23350a94-a70f-4322-8fad-a5ddf5152cf4-agent-certs\") pod \"konnectivity-agent-9c6rx\" (UID: \"23350a94-a70f-4322-8fad-a5ddf5152cf4\") " pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-sys\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-k8s-cni-cncf-io\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-cni-bin\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-env-overrides\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-daemon-config\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovn-node-metrics-cert\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-registration-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e2a7c85-b9e5-4879-a00d-d73018cf4252-registration-dir\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.124288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.123810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.155232 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.155192    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 12:14:27.158969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.158947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfh72\" (UniqueName: \"kubernetes.io/projected/e7a698e7-ea0e-4c2b-afba-7c153c612e02-kube-api-access-dfh72\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.159048 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.159000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckrc\" (UniqueName: \"kubernetes.io/projected/9e2a7c85-b9e5-4879-a00d-d73018cf4252-kube-api-access-vckrc\") pod \"aws-ebs-csi-driver-node-ns2xm\" (UID: \"9e2a7c85-b9e5-4879-a00d-d73018cf4252\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.164532 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.164505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrr9v\" (UniqueName: \"kubernetes.io/projected/f3ff7d73-8d6d-4152-98bc-cb8625f31cd7-kube-api-access-jrr9v\") pod \"multus-additional-cni-plugins-jlgq7\" (UID: \"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7\") " pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.173858 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.173833    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6gsq"
Sep 11 12:14:27.180149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.180120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5z6\" (UniqueName: \"kubernetes.io/projected/2dfc139d-214b-4e2f-8424-f97bc38788a0-kube-api-access-5v5z6\") pod \"node-ca-6b29g\" (UID: \"2dfc139d-214b-4e2f-8424-f97bc38788a0\") " pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.202128 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.202092    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6gsq"
Sep 11 12:14:27.223750 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.223716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysconfig\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.223750 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.223754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-cnibin\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.224536 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-conf-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.224686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysconfig\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.224686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5nw\" (UniqueName: \"kubernetes.io/projected/61b4af8a-0798-441f-84b5-4c24cb5e3839-kube-api-access-xj5nw\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.224686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-host\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.224686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-cnibin\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.224686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-etc-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.224686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-conf-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-etc-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87a282b2-6cba-47e6-8831-de744de9a5af-kubelet-config\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-host-slash\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-kubernetes\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-node-log\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224871    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-host\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87a282b2-6cba-47e6-8831-de744de9a5af-kubelet-config\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.224841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-cni-netd\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-host-slash\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.225046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovnkube-config\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23350a94-a70f-4322-8fad-a5ddf5152cf4-agent-certs\") pod \"konnectivity-agent-9c6rx\" (UID: \"23350a94-a70f-4322-8fad-a5ddf5152cf4\") " pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-sys\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-node-log\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-kubernetes\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-cni-netd\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-k8s-cni-cncf-io\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-cni-bin\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-env-overrides\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-daemon-config\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovn-node-metrics-cert\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-kubelet\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.225563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-modprobe-d\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysctl-d\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovnkube-script-lib\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-run\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-socket-dir-parent\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmx7\" (UniqueName: \"kubernetes.io/projected/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-kube-api-access-6kmx7\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-systemd\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-cni-bin\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-var-lib-kubelet\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.225957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcttw\" (UniqueName: \"kubernetes.io/projected/36901f08-aeb6-4775-b8f3-a8057c5afe33-kube-api-access-lcttw\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-cni-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovnkube-config\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-cni-bin\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-sys\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-multus-certs\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-socket-dir-parent\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-k8s-cni-cncf-io\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-multus-certs\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.226316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysctl-d\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-systemd\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-var-lib-kubelet\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovnkube-script-lib\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-run\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.226816    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysctl-conf\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.226869    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:14:27.726855265 +0000 UTC m=+3.117392586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-etc-kubernetes\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-cni-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.226942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/041a90da-ab4a-47b2-b4b5-463f9a448d17-tmp\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-cni-bin\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-systemd-units\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-slash\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-run-netns\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61b4af8a-0798-441f-84b5-4c24cb5e3839-cni-binary-copy\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-kubelet\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-systemd-units\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-slash\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-sysctl-conf\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-iptables-alerter-script\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-modprobe-d\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-run-netns\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-tuned\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-log-socket\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63bae1b4-810c-4092-a4ba-cc816e956adc-hosts-file\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-system-cni-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-cni-multus\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227675    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xvb\" (UniqueName: \"kubernetes.io/projected/041a90da-ab4a-47b2-b4b5-463f9a448d17-kube-api-access-87xvb\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-kubelet\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-systemd\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.227995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-ovn\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-run-ovn-kubernetes\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36901f08-aeb6-4775-b8f3-a8057c5afe33-env-overrides\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23350a94-a70f-4322-8fad-a5ddf5152cf4-konnectivity-ca\") pod \"konnectivity-agent-9c6rx\" (UID: \"23350a94-a70f-4322-8fad-a5ddf5152cf4\") " pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87a282b2-6cba-47e6-8831-de744de9a5af-dbus\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bae1b4-810c-4092-a4ba-cc816e956adc-tmp-dir\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5ph\" (UniqueName: \"kubernetes.io/projected/63bae1b4-810c-4092-a4ba-cc816e956adc-kube-api-access-vr5ph\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-netns\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.227972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-lib-modules\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-var-lib-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61b4af8a-0798-441f-84b5-4c24cb5e3839-cni-binary-copy\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-hostroot\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-kubelet\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-systemd\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-run-ovn\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-run-ovn-kubernetes\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.228821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-iptables-alerter-script\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/23350a94-a70f-4322-8fad-a5ddf5152cf4-konnectivity-ca\") pod \"konnectivity-agent-9c6rx\" (UID: \"23350a94-a70f-4322-8fad-a5ddf5152cf4\") " pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.228962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-etc-kubernetes\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-system-cni-dir\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87a282b2-6cba-47e6-8831-de744de9a5af-dbus\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-log-socket\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-run-netns\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/041a90da-ab4a-47b2-b4b5-463f9a448d17-lib-modules\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-var-lib-openvswitch\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36901f08-aeb6-4775-b8f3-a8057c5afe33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63bae1b4-810c-4092-a4ba-cc816e956adc-hosts-file\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.229665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bae1b4-810c-4092-a4ba-cc816e956adc-tmp-dir\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.230181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229742    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-hostroot\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.230181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/61b4af8a-0798-441f-84b5-4c24cb5e3839-host-var-lib-cni-multus\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.230181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/61b4af8a-0798-441f-84b5-4c24cb5e3839-multus-daemon-config\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.230181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.229942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/041a90da-ab4a-47b2-b4b5-463f9a448d17-tmp\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.230373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.230213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/041a90da-ab4a-47b2-b4b5-463f9a448d17-etc-tuned\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.232941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.232906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/23350a94-a70f-4322-8fad-a5ddf5152cf4-agent-certs\") pod \"konnectivity-agent-9c6rx\" (UID: \"23350a94-a70f-4322-8fad-a5ddf5152cf4\") " pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.234331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.234230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36901f08-aeb6-4775-b8f3-a8057c5afe33-ovn-node-metrics-cert\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.243997 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.243917    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:27.243997 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.243948    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:27.243997 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.243964    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:27.244246 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.244044    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:27.744022119 +0000 UTC m=+3.134559465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:27.255973 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.255945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xvb\" (UniqueName: \"kubernetes.io/projected/041a90da-ab4a-47b2-b4b5-463f9a448d17-kube-api-access-87xvb\") pod \"tuned-stnqf\" (UID: \"041a90da-ab4a-47b2-b4b5-463f9a448d17\") " pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.257350 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.257291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcttw\" (UniqueName: \"kubernetes.io/projected/36901f08-aeb6-4775-b8f3-a8057c5afe33-kube-api-access-lcttw\") pod \"ovnkube-node-5szwd\" (UID: \"36901f08-aeb6-4775-b8f3-a8057c5afe33\") " pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.259024 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.258995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5nw\" (UniqueName: \"kubernetes.io/projected/61b4af8a-0798-441f-84b5-4c24cb5e3839-kube-api-access-xj5nw\") pod \"multus-72m7f\" (UID: \"61b4af8a-0798-441f-84b5-4c24cb5e3839\") " pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.270356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.270326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmx7\" (UniqueName: \"kubernetes.io/projected/0e4c1f98-c509-487f-a0b4-1ffb73fde80a-kube-api-access-6kmx7\") pod \"iptables-alerter-cbhkm\" (UID: \"0e4c1f98-c509-487f-a0b4-1ffb73fde80a\") " pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.301127 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.301087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5ph\" (UniqueName: \"kubernetes.io/projected/63bae1b4-810c-4092-a4ba-cc816e956adc-kube-api-access-vr5ph\") pod \"node-resolver-mhljr\" (UID: \"63bae1b4-810c-4092-a4ba-cc816e956adc\") " pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.309286 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.309213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6b29g"
Sep 11 12:14:27.315150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.315120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm"
Sep 11 12:14:27.325669 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.325438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jlgq7"
Sep 11 12:14:27.327800 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.327746    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfc139d_214b_4e2f_8424_f97bc38788a0.slice/crio-7c76a9bc43ef64b4eba0bcc8f268adb8b3d869d262651fd4577e5d2ae0ebfe35 WatchSource:0}: Error finding container 7c76a9bc43ef64b4eba0bcc8f268adb8b3d869d262651fd4577e5d2ae0ebfe35: Status 404 returned error can't find the container with id 7c76a9bc43ef64b4eba0bcc8f268adb8b3d869d262651fd4577e5d2ae0ebfe35
Sep 11 12:14:27.329265 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.329239    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2a7c85_b9e5_4879_a00d_d73018cf4252.slice/crio-0366badc7e6af5555cb79c2459bc25567fb1584f22fbe98b849d051c3dc9c54b WatchSource:0}: Error finding container 0366badc7e6af5555cb79c2459bc25567fb1584f22fbe98b849d051c3dc9c54b: Status 404 returned error can't find the container with id 0366badc7e6af5555cb79c2459bc25567fb1584f22fbe98b849d051c3dc9c54b
Sep 11 12:14:27.332524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.332237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mhljr"
Sep 11 12:14:27.336225 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.336192    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ff7d73_8d6d_4152_98bc_cb8625f31cd7.slice/crio-1b2935cc55b810c6ee072038c3ac63a9aa68ef46a45dc384389714ea432db4a7 WatchSource:0}: Error finding container 1b2935cc55b810c6ee072038c3ac63a9aa68ef46a45dc384389714ea432db4a7: Status 404 returned error can't find the container with id 1b2935cc55b810c6ee072038c3ac63a9aa68ef46a45dc384389714ea432db4a7
Sep 11 12:14:27.338807 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.338759    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cbhkm"
Sep 11 12:14:27.345077 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.344907    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stnqf"
Sep 11 12:14:27.351620 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.351597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-72m7f"
Sep 11 12:14:27.353062 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.353037    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4c1f98_c509_487f_a0b4_1ffb73fde80a.slice/crio-1b1c5ac3b7f6af803b510fd3ef5bf0f1a181e54ddfdd189fed85c5205eb5860e WatchSource:0}: Error finding container 1b1c5ac3b7f6af803b510fd3ef5bf0f1a181e54ddfdd189fed85c5205eb5860e: Status 404 returned error can't find the container with id 1b1c5ac3b7f6af803b510fd3ef5bf0f1a181e54ddfdd189fed85c5205eb5860e
Sep 11 12:14:27.357140 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.357109    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041a90da_ab4a_47b2_b4b5_463f9a448d17.slice/crio-3ea4c51cc5d539bc6ff23982acf7751797c9d9f4ccfe383e86aa80de0a93c5ea WatchSource:0}: Error finding container 3ea4c51cc5d539bc6ff23982acf7751797c9d9f4ccfe383e86aa80de0a93c5ea: Status 404 returned error can't find the container with id 3ea4c51cc5d539bc6ff23982acf7751797c9d9f4ccfe383e86aa80de0a93c5ea
Sep 11 12:14:27.357301 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.357284    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:27.361921 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.361895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:27.367158 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.367126    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b4af8a_0798_441f_84b5_4c24cb5e3839.slice/crio-782352e24e959ee70ba3cc3869be70cd07d0cf638302f8c9fc34b2b780660a7f WatchSource:0}: Error finding container 782352e24e959ee70ba3cc3869be70cd07d0cf638302f8c9fc34b2b780660a7f: Status 404 returned error can't find the container with id 782352e24e959ee70ba3cc3869be70cd07d0cf638302f8c9fc34b2b780660a7f
Sep 11 12:14:27.373283 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.373254    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36901f08_aeb6_4775_b8f3_a8057c5afe33.slice/crio-3125e5b1950ef612f2427ec23046dd4675af9fb8586ec88d1b4f85e3ceed51e1 WatchSource:0}: Error finding container 3125e5b1950ef612f2427ec23046dd4675af9fb8586ec88d1b4f85e3ceed51e1: Status 404 returned error can't find the container with id 3125e5b1950ef612f2427ec23046dd4675af9fb8586ec88d1b4f85e3ceed51e1
Sep 11 12:14:27.377724 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:14:27.377697    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23350a94_a70f_4322_8fad_a5ddf5152cf4.slice/crio-00a3e0ac5f7762ad120c2742105d56c8ee67b50f04b6595027f8a8015e0ff366 WatchSource:0}: Error finding container 00a3e0ac5f7762ad120c2742105d56c8ee67b50f04b6595027f8a8015e0ff366: Status 404 returned error can't find the container with id 00a3e0ac5f7762ad120c2742105d56c8ee67b50f04b6595027f8a8015e0ff366
Sep 11 12:14:27.630266 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.630174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:27.630469 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.630313    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:27.630469 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.630389    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:28.630369808 +0000 UTC m=+4.020907143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:27.730976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.730930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:27.731185 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.731116    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:27.731260 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.731204    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:14:28.731182969 +0000 UTC m=+4.121720307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:27.831741 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:27.831700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:27.831947 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.831903    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:27.831947 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.831928    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:27.831947 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.831938    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:27.832123 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:27.832003    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:28.831985307 +0000 UTC m=+4.222522649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:28.111267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.111168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stnqf" event={"ID":"041a90da-ab4a-47b2-b4b5-463f9a448d17","Type":"ContainerStarted","Data":"3ea4c51cc5d539bc6ff23982acf7751797c9d9f4ccfe383e86aa80de0a93c5ea"}
Sep 11 12:14:28.112893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.112855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cbhkm" event={"ID":"0e4c1f98-c509-487f-a0b4-1ffb73fde80a","Type":"ContainerStarted","Data":"1b1c5ac3b7f6af803b510fd3ef5bf0f1a181e54ddfdd189fed85c5205eb5860e"}
Sep 11 12:14:28.114867 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.114834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhljr" event={"ID":"63bae1b4-810c-4092-a4ba-cc816e956adc","Type":"ContainerStarted","Data":"b9eaebaafb6dcc48a18ebe4fca4fb21312339c3dd06763029367460b457016bd"}
Sep 11 12:14:28.116865 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.116830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6b29g" event={"ID":"2dfc139d-214b-4e2f-8424-f97bc38788a0","Type":"ContainerStarted","Data":"7c76a9bc43ef64b4eba0bcc8f268adb8b3d869d262651fd4577e5d2ae0ebfe35"}
Sep 11 12:14:28.118533 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.118417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"3125e5b1950ef612f2427ec23046dd4675af9fb8586ec88d1b4f85e3ceed51e1"}
Sep 11 12:14:28.120558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.120528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72m7f" event={"ID":"61b4af8a-0798-441f-84b5-4c24cb5e3839","Type":"ContainerStarted","Data":"782352e24e959ee70ba3cc3869be70cd07d0cf638302f8c9fc34b2b780660a7f"}
Sep 11 12:14:28.122020 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.121994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerStarted","Data":"1b2935cc55b810c6ee072038c3ac63a9aa68ef46a45dc384389714ea432db4a7"}
Sep 11 12:14:28.123404 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.123380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm" event={"ID":"9e2a7c85-b9e5-4879-a00d-d73018cf4252","Type":"ContainerStarted","Data":"0366badc7e6af5555cb79c2459bc25567fb1584f22fbe98b849d051c3dc9c54b"}
Sep 11 12:14:28.124950 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.124884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9c6rx" event={"ID":"23350a94-a70f-4322-8fad-a5ddf5152cf4","Type":"ContainerStarted","Data":"00a3e0ac5f7762ad120c2742105d56c8ee67b50f04b6595027f8a8015e0ff366"}
Sep 11 12:14:28.206161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.206108    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:09:27 +0000 UTC" deadline="2028-04-28 10:52:21.923330253 +0000 UTC"
Sep 11 12:14:28.206161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.206155    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14278h37m53.717179792s"
Sep 11 12:14:28.639047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.639004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:28.639467 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.639169    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:28.639467 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.639263    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:30.639241542 +0000 UTC m=+6.029778881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:28.739895 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.739764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:28.740094 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.739926    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:28.740094 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.740016    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:14:30.739996357 +0000 UTC m=+6.130533683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:28.840655 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:28.840604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:28.840866 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.840836    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:28.840866 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.840859    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:28.840988 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.840874    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:28.840988 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:28.840941    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:30.840920354 +0000 UTC m=+6.231457695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:29.115739 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:29.114723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:29.115739 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:29.115120    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:29.115739 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:29.115589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:29.115739 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:29.115686    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:29.117710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:29.116501    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:29.117710 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:29.116620    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:29.207322 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:29.207202    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:09:27 +0000 UTC" deadline="2028-03-20 01:39:28.698442417 +0000 UTC"
Sep 11 12:14:29.207322 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:29.207266    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13333h24m59.491179778s"
Sep 11 12:14:30.145000 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:30.142988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal" event={"ID":"243f33b0da41e521566de787222b729f","Type":"ContainerStarted","Data":"48abc2700ce1558b55e9465cb9a76ba407893ef19cb1b214ec88bc2609a1e4a6"}
Sep 11 12:14:30.661966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:30.661924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:30.662167 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.662097    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:30.662237 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.662175    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:34.662154525 +0000 UTC m=+10.052691853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:30.763561 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:30.763054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:30.763561 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.763207    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:30.763561 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.763276    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:14:34.763255795 +0000 UTC m=+10.153793143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:30.864749 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:30.863953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:30.864749 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.864123    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:30.864749 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.864143    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:30.864749 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.864157    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:30.864749 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:30.864305    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:34.864276049 +0000 UTC m=+10.254813389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:31.103179 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:31.103083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:31.103349 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:31.103228    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:31.103687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:31.103646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:31.103808 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:31.103763    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:31.103884 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:31.103817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:31.103935 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:31.103892    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:31.153993 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:31.152583    2561 generic.go:356] "Generic (PLEG): container finished" podID="ec3255f401d21cf30d3c48f303e873f0" containerID="5aada57e724b198ec67995be5ffabb41ba8a21980e86e6824532b74b7eb0f99a" exitCode=0
Sep 11 12:14:31.153993 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:31.153657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerDied","Data":"5aada57e724b198ec67995be5ffabb41ba8a21980e86e6824532b74b7eb0f99a"}
Sep 11 12:14:31.179497 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:31.179405    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal" podStartSLOduration=4.179378424 podStartE2EDuration="4.179378424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:30.174466918 +0000 UTC m=+5.565004265" watchObservedRunningTime="2026-09-11 12:14:31.179378424 +0000 UTC m=+6.569915770"
Sep 11 12:14:33.101657 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:33.101619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:33.102186 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:33.101796    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:33.102440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:33.102266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:33.102440 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:33.102369    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:33.102910 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:33.102747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:33.102910 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:33.102851    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:34.696832 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:34.696762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:34.697341 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.696914    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:34.697341 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.697001    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:42.696976888 +0000 UTC m=+18.087514238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:34.797681 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:34.797624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:34.797866 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.797797    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:34.797930 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.797871    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:14:42.797848235 +0000 UTC m=+18.188385576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:34.898614 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:34.898548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:34.898825 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.898756    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:34.898825 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.898777    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:34.898825 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.898788    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:34.899064 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:34.898847    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:42.898832481 +0000 UTC m=+18.289369817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:35.102721 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:35.101830    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:35.102721 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:35.101964    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:35.102721 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:35.102020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:35.102721 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:35.102092    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:35.102721 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:35.102535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:35.102721 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:35.102627    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:37.101072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:37.100987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:37.101072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:37.101020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:37.101633 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:37.100987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:37.101633 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:37.101144    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:37.101633 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:37.101251    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:37.101633 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:37.101365    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:39.101047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:39.101010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:39.101527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:39.101057    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:39.101527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:39.101014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:39.101527 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:39.101127    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:39.101527 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:39.101191    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:39.101527 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:39.101258    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:41.101162 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:41.101127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:41.101162 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:41.101165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:41.101649 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:41.101132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:41.101649 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:41.101257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:41.102303 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:41.102271    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:41.102429 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:41.102077    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:42.756450 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:42.756417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:42.756814 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.756554    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:42.756814 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.756608    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.756592337 +0000 UTC m=+34.147129666 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:42.856803 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:42.856769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:42.856977 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.856925    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:42.857023 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.856989    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.856974907 +0000 UTC m=+34.247512231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:42.957502 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:42.957468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:42.957673 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.957611    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:42.957673 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.957626    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:42.957673 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.957636    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:42.957855 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:42.957697    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.957679322 +0000 UTC m=+34.348216659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:43.100504 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:43.100415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:43.100504 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:43.100438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:43.100707 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:43.100544    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:43.100707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:43.100579    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:43.100707 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:43.100696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:43.100858 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:43.100781    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:45.102050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:45.101714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:45.102050 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:45.101823    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:45.102050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:45.101913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:45.102050 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:45.102032    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:45.102717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:45.102086    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:45.102717 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:45.102171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:47.100613 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.100586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:47.101142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.100621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:47.101142 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:47.100728    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:47.101142 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:47.100768    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:47.101142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.100788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:47.101142 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:47.100854    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:47.189238 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.189201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stnqf" event={"ID":"041a90da-ab4a-47b2-b4b5-463f9a448d17","Type":"ContainerStarted","Data":"891ecf30df5c15d6b91f906f2502e684504a682dee9c8903caf478f3e38a2871"}
Sep 11 12:14:47.190994 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.190965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mhljr" event={"ID":"63bae1b4-810c-4092-a4ba-cc816e956adc","Type":"ContainerStarted","Data":"7d85cae055d47e73b558f0cdd3c0d2de86c2e753505ce66f338559542963d163"}
Sep 11 12:14:47.192713 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.192687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6b29g" event={"ID":"2dfc139d-214b-4e2f-8424-f97bc38788a0","Type":"ContainerStarted","Data":"97184ee95edd6ac2902e8c74bad5249f8ee10842a7ca0e19c20aa1adfa990062"}
Sep 11 12:14:47.197966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.197942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"a1959196e56241aebfee9baa4030ab85e48545217f3e6e4d66744e5d46825869"}
Sep 11 12:14:47.198142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.198002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"35e67782d8fbd4b67362f65b4077a131f53786be56a431287e0402d3b5b804c6"}
Sep 11 12:14:47.198142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.198031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"2606370202fe02dd0f88fba396df45ee2b1e318142cca04ab39f9d04744e1fd1"}
Sep 11 12:14:47.199363 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.199342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-72m7f" event={"ID":"61b4af8a-0798-441f-84b5-4c24cb5e3839","Type":"ContainerStarted","Data":"778916f97f99dd0d44f34a6a6945b6918fda1c382caa74e9b86500a8673fc109"}
Sep 11 12:14:47.201155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.201128    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3ff7d73-8d6d-4152-98bc-cb8625f31cd7" containerID="aca55567cc9fad1589ddce824309156094f934cbd40fee8966535d603e4dba95" exitCode=0
Sep 11 12:14:47.201226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.201200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerDied","Data":"aca55567cc9fad1589ddce824309156094f934cbd40fee8966535d603e4dba95"}
Sep 11 12:14:47.202741 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.202719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm" event={"ID":"9e2a7c85-b9e5-4879-a00d-d73018cf4252","Type":"ContainerStarted","Data":"c51cbf4ee01506f2e89a0ef51cf59954a7be0091b1d13456c8becd51ba4f2364"}
Sep 11 12:14:47.204853 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.204803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerStarted","Data":"2178f4e89fc3619a8dac6307a3aefaac3339d8ce05859cd04f80089514b827e7"}
Sep 11 12:14:47.206319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.206297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9c6rx" event={"ID":"23350a94-a70f-4322-8fad-a5ddf5152cf4","Type":"ContainerStarted","Data":"36504d1437c56b89f0a886cce2b93a9bcbb560316a02334003a323187f0163b4"}
Sep 11 12:14:47.208202 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.208166    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-stnqf" podStartSLOduration=1.764979808 podStartE2EDuration="21.20815522s" totalImagesPullingTime="19.443175412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:47.207879963 +0000 UTC m=+22.598417306" watchObservedRunningTime="2026-09-11 12:14:47.20815522 +0000 UTC m=+22.598692561"
Sep 11 12:14:47.248186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.248142    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mhljr" podStartSLOduration=2.18281374 podStartE2EDuration="21.248125705s" totalImagesPullingTime="19.065311965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:47.247981954 +0000 UTC m=+22.638519297" watchObservedRunningTime="2026-09-11 12:14:47.248125705 +0000 UTC m=+22.638663045"
Sep 11 12:14:47.265716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.265672    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-72m7f" podStartSLOduration=1.780976991 podStartE2EDuration="21.265653732s" totalImagesPullingTime="19.484676741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:47.265240155 +0000 UTC m=+22.655777496" watchObservedRunningTime="2026-09-11 12:14:47.265653732 +0000 UTC m=+22.656191074"
Sep 11 12:14:47.283194 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.283153    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9c6rx" podStartSLOduration=1.859977345 podStartE2EDuration="21.283138049s" totalImagesPullingTime="19.423160704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:47.282783812 +0000 UTC m=+22.673321156" watchObservedRunningTime="2026-09-11 12:14:47.283138049 +0000 UTC m=+22.673675390"
Sep 11 12:14:47.298599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.298554    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" podStartSLOduration=20.298539253 podStartE2EDuration="20.298539253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:47.298057971 +0000 UTC m=+22.688595312" watchObservedRunningTime="2026-09-11 12:14:47.298539253 +0000 UTC m=+22.689076595"
Sep 11 12:14:47.312378 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:47.312334    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6b29g" podStartSLOduration=1.8429061610000002 podStartE2EDuration="21.312316972s" totalImagesPullingTime="19.469410811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:47.312145795 +0000 UTC m=+22.702683137" watchObservedRunningTime="2026-09-11 12:14:47.312316972 +0000 UTC m=+22.702854315"
Sep 11 12:14:48.213346 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:48.213305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"229818f7b43c1f86f7b0e068a2a936c3592cac952b797c985f770e4e52ee9aee"}
Sep 11 12:14:48.214225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:48.213489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"a6a0427adc1b897b1b8ffc0328aca0554c2ecaa2159a669b0d9664f780503275"}
Sep 11 12:14:48.214225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:48.213547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"8c9bad2e0a61d7f5d85b4932f5bef74a325b4c38e51383e5fb80617164ece2b8"}
Sep 11 12:14:48.291664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:48.291632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:48.292421 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:48.292378    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:48.307021 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:48.306998    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 12:14:49.084989 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.084871    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T12:14:48.307016061Z","UUID":"3dc0919d-b6e0-4337-a082-8ac8db2fc503","Handler":null,"Name":"","Endpoint":""}
Sep 11 12:14:49.086575 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.086553    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 12:14:49.086699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.086588    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 12:14:49.100421 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.100377    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:49.100421 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.100392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:49.100613 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.100376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:49.100613 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:49.100516    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:49.100709 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:49.100625    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:49.100760 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:49.100706    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:49.217512 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.217475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cbhkm" event={"ID":"0e4c1f98-c509-487f-a0b4-1ffb73fde80a","Type":"ContainerStarted","Data":"93c9f77fa92d4ac5df2ddcf86daa055a3b5b77264199328c2387e6cc4017f01f"}
Sep 11 12:14:49.219680 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.219649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm" event={"ID":"9e2a7c85-b9e5-4879-a00d-d73018cf4252","Type":"ContainerStarted","Data":"50c3eaf158cf55e18bfeffd899b806593ee47fa30bc3e65a69217bdac0e0ef1f"}
Sep 11 12:14:49.219861 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.219845    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:49.220501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.220481    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9c6rx"
Sep 11 12:14:49.270791 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:49.270374    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cbhkm" podStartSLOduration=4.213425086 podStartE2EDuration="23.270359368s" totalImagesPullingTime="19.056934282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:49.247327616 +0000 UTC m=+24.637864959" watchObservedRunningTime="2026-09-11 12:14:49.270359368 +0000 UTC m=+24.660896709"
Sep 11 12:14:50.226060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:50.225848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"16e948fca76e2e6a581b962fd134797ab1f31e5b0eae83d2af242de099035603"}
Sep 11 12:14:50.228229 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:50.228201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm" event={"ID":"9e2a7c85-b9e5-4879-a00d-d73018cf4252","Type":"ContainerStarted","Data":"2da2dc4c150df6f7b839a48f1388c87ea4967576159b465702f65c771c7ea970"}
Sep 11 12:14:51.100515 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:51.100463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:51.100515 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:51.100522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:51.100800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:51.100463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:51.100800 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:51.100602    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:51.100800 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:51.100645    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:51.100800 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:51.100721    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:52.236550 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.236517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" event={"ID":"36901f08-aeb6-4775-b8f3-a8057c5afe33","Type":"ContainerStarted","Data":"e6213d3e11916156ff7c901f5eb39fc945a8e731a3bcf28ba30ddb2c34576866"}
Sep 11 12:14:52.237097 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.236743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:52.238547 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.238519    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3ff7d73-8d6d-4152-98bc-cb8625f31cd7" containerID="2812bef1f2d4ff72159c7c94eddfcd7b28294b725d4990bb59ea31401b5f4a43" exitCode=0
Sep 11 12:14:52.238650 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.238563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerDied","Data":"2812bef1f2d4ff72159c7c94eddfcd7b28294b725d4990bb59ea31401b5f4a43"}
Sep 11 12:14:52.251914 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.251895    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:52.267573 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.267525    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd" podStartSLOduration=6.825449439 podStartE2EDuration="26.267510761s" totalImagesPullingTime="19.442061322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:52.267205617 +0000 UTC m=+27.657742960" watchObservedRunningTime="2026-09-11 12:14:52.267510761 +0000 UTC m=+27.658048104"
Sep 11 12:14:52.267742 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:52.267724    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ns2xm" podStartSLOduration=4.2455592 podStartE2EDuration="26.267718296s" totalImagesPullingTime="22.022159096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:50.261265089 +0000 UTC m=+25.651802431" watchObservedRunningTime="2026-09-11 12:14:52.267718296 +0000 UTC m=+27.658256031"
Sep 11 12:14:53.105635 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.105598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:53.105801 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.105598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:53.105801 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.105729    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:53.105801 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:53.105723    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:53.105956 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:53.105825    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:53.105956 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:53.105898    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:53.241303 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.241276    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:53.241713 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.241366    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:53.255665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.255642    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:14:53.679462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.679377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2cbzp"]
Sep 11 12:14:53.679603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.679507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:53.679645 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:53.679616    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:53.682679 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.682652    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vb9f7"]
Sep 11 12:14:53.682820 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.682756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:53.682885 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:53.682859    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:53.683252 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.683230    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4zck"]
Sep 11 12:14:53.683354 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:53.683315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:53.683528 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:53.683491    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:54.245217 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:54.245185    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3ff7d73-8d6d-4152-98bc-cb8625f31cd7" containerID="7a23dc08075713fd1cf35437046ceefb94eb76d547aafd4936632bbda4b4ae5c" exitCode=0
Sep 11 12:14:54.245912 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:54.245270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerDied","Data":"7a23dc08075713fd1cf35437046ceefb94eb76d547aafd4936632bbda4b4ae5c"}
Sep 11 12:14:55.103196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:55.103020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:55.103323 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:55.103089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:55.103323 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:55.103270    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:55.103416 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:55.103381    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:55.103464 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:55.103104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:55.103523 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:55.103506    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:55.249280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:55.249247    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3ff7d73-8d6d-4152-98bc-cb8625f31cd7" containerID="2ee65bfc0fce68c5676017f6844bd1686d998613a21c4945b92da2577c9f0db3" exitCode=0
Sep 11 12:14:55.249678 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:55.249331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerDied","Data":"2ee65bfc0fce68c5676017f6844bd1686d998613a21c4945b92da2577c9f0db3"}
Sep 11 12:14:57.087790 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:57.087746    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:14:57.100715 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:57.100670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:57.100863 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:57.100670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:57.100863 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:57.100805    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:57.100863 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:57.100677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:57.101004 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:57.100882    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:57.101004 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:57.100946    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:57.190881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:57.190841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:14:58.773303 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:58.773269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:58.773908 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.773411    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:58.773908 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.773486    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs podName:e7a698e7-ea0e-4c2b-afba-7c153c612e02 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:30.773470295 +0000 UTC m=+66.164007619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs") pod "network-metrics-daemon-vb9f7" (UID: "e7a698e7-ea0e-4c2b-afba-7c153c612e02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:58.873905 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:58.873863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:58.874091 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.874013    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:58.874091 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.874087    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret podName:87a282b2-6cba-47e6-8831-de744de9a5af nodeName:}" failed. No retries permitted until 2026-09-11 12:15:30.874069132 +0000 UTC m=+66.264606465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret") pod "global-pull-secret-syncer-2cbzp" (UID: "87a282b2-6cba-47e6-8831-de744de9a5af") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:58.975217 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:58.975181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:58.975427 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.975357    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:58.975427 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.975382    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:58.975427 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.975411    2561 projected.go:196] Error preparing data for projected volume kube-api-access-g7qzm for pod openshift-network-diagnostics/network-check-target-j4zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:58.975562 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:58.975474    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm podName:57232506-b657-42a9-b59b-df156dfae7f3 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:30.975455199 +0000 UTC m=+66.365992530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7qzm" (UniqueName: "kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm") pod "network-check-target-j4zck" (UID: "57232506-b657-42a9-b59b-df156dfae7f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:59.101074 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.101043    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:14:59.101074 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.101058    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:14:59.101321 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:59.101175    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2cbzp" podUID="87a282b2-6cba-47e6-8831-de744de9a5af"
Sep 11 12:14:59.101321 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:59.101244    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9f7" podUID="e7a698e7-ea0e-4c2b-afba-7c153c612e02"
Sep 11 12:14:59.101321 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.101292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:14:59.101493 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:14:59.101361    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j4zck" podUID="57232506-b657-42a9-b59b-df156dfae7f3"
Sep 11 12:14:59.890663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.890630    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeReady"
Sep 11 12:14:59.891028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.890819    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 12:14:59.978449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.978323    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p2rq7"]
Sep 11 12:14:59.993102 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.993065    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rbh52"]
Sep 11 12:14:59.993239 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.993186    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:14:59.995760 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.995606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 12:14:59.995760 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.995638    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgchp\""
Sep 11 12:14:59.995962 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:14:59.995912    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 12:15:00.007452 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.007425    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"]
Sep 11 12:15:00.007579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.007481    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.011426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.011386    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 12:15:00.011670 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.011654    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ld655\""
Sep 11 12:15:00.012066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.012048    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 12:15:00.012174 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.012115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 12:15:00.025205 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.025172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2rq7"]
Sep 11 12:15:00.025342 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.025221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbh52"]
Sep 11 12:15:00.025342 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.025242    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"]
Sep 11 12:15:00.025342 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.025247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.028609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.028588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 12:15:00.028988 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.028970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 12:15:00.029177 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.029162    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bmqgw\""
Sep 11 12:15:00.029304 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.029290    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 12:15:00.035612 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.035594    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 12:15:00.083515 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.083488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5628a9-e957-4057-a70e-08d721e93717-cert\") pod \"ingress-canary-rbh52\" (UID: \"2b5628a9-e957-4057-a70e-08d721e93717\") " pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.083646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.083527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcrc\" (UniqueName: \"kubernetes.io/projected/2b5628a9-e957-4057-a70e-08d721e93717-kube-api-access-6tcrc\") pod \"ingress-canary-rbh52\" (UID: \"2b5628a9-e957-4057-a70e-08d721e93717\") " pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.083646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.083570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.083646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.083594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3f287d-baa3-45f9-9108-429e7bf73b9a-tmp-dir\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.083646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.083614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbdv\" (UniqueName: \"kubernetes.io/projected/8f3f287d-baa3-45f9-9108-429e7bf73b9a-kube-api-access-tcbdv\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.083646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.083628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3f287d-baa3-45f9-9108-429e7bf73b9a-config-volume\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.184838 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.184798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.184838 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.184840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-installation-pull-secrets\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.184865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcrc\" (UniqueName: \"kubernetes.io/projected/2b5628a9-e957-4057-a70e-08d721e93717-kube-api-access-6tcrc\") pod \"ingress-canary-rbh52\" (UID: \"2b5628a9-e957-4057-a70e-08d721e93717\") " pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.185051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.184916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.185051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.184932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3f287d-baa3-45f9-9108-429e7bf73b9a-tmp-dir\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.185051 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.185045    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:00.185212 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-bound-sa-token\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185212 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbdv\" (UniqueName: \"kubernetes.io/projected/8f3f287d-baa3-45f9-9108-429e7bf73b9a-kube-api-access-tcbdv\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.185212 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.185108    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls podName:8f3f287d-baa3-45f9-9108-429e7bf73b9a nodeName:}" failed. No retries permitted until 2026-09-11 12:15:00.685088941 +0000 UTC m=+36.075626281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls") pod "dns-default-p2rq7" (UID: "8f3f287d-baa3-45f9-9108-429e7bf73b9a") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:00.185212 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3f287d-baa3-45f9-9108-429e7bf73b9a-config-volume\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.185212 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-image-registry-private-configuration\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-certificates\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-trusted-ca\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3f287d-baa3-45f9-9108-429e7bf73b9a-tmp-dir\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.185469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-ca-trust-extracted\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6tf5\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-kube-api-access-r6tf5\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.185469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5628a9-e957-4057-a70e-08d721e93717-cert\") pod \"ingress-canary-rbh52\" (UID: \"2b5628a9-e957-4057-a70e-08d721e93717\") " pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.185684 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.185666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3f287d-baa3-45f9-9108-429e7bf73b9a-config-volume\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.189731 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.189605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5628a9-e957-4057-a70e-08d721e93717-cert\") pod \"ingress-canary-rbh52\" (UID: \"2b5628a9-e957-4057-a70e-08d721e93717\") " pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.196971 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.196949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcrc\" (UniqueName: \"kubernetes.io/projected/2b5628a9-e957-4057-a70e-08d721e93717-kube-api-access-6tcrc\") pod \"ingress-canary-rbh52\" (UID: \"2b5628a9-e957-4057-a70e-08d721e93717\") " pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.198594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.198573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbdv\" (UniqueName: \"kubernetes.io/projected/8f3f287d-baa3-45f9-9108-429e7bf73b9a-kube-api-access-tcbdv\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.265356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.265277    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3ff7d73-8d6d-4152-98bc-cb8625f31cd7" containerID="51f60838b107c7532ae950116d9f75956a2acb3367e1d22df1e839fc6ef76a04" exitCode=0
Sep 11 12:15:00.265356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.265325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerDied","Data":"51f60838b107c7532ae950116d9f75956a2acb3367e1d22df1e839fc6ef76a04"}
Sep 11 12:15:00.273595 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.273565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:00.286219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-ca-trust-extracted\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286320 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6tf5\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-kube-api-access-r6tf5\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286320 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286320 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-installation-pull-secrets\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286560 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.286392    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 12:15:00.286560 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.286430    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9bcfb56b-4z6m6: secret "image-registry-tls" not found
Sep 11 12:15:00.286560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-bound-sa-token\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286560 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.286491    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls podName:3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:00.786463349 +0000 UTC m=+36.177000670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls") pod "image-registry-7d9bcfb56b-4z6m6" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee") : secret "image-registry-tls" not found
Sep 11 12:15:00.286560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-image-registry-private-configuration\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286815 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-certificates\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286815 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-ca-trust-extracted\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.286815 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.286699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-trusted-ca\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.287196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.287176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-certificates\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.287752 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.287735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-trusted-ca\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.288599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.288582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-installation-pull-secrets\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.288695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.288678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-image-registry-private-configuration\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.298167 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.298149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6tf5\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-kube-api-access-r6tf5\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.315459 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.315415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbh52"
Sep 11 12:15:00.315891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.315870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-bound-sa-token\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.338519 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.338484    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:00.507823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.507789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:00.635010 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.634969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbh52"]
Sep 11 12:15:00.639892 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:00.639861    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5628a9_e957_4057_a70e_08d721e93717.slice/crio-64db6bf74c50986c7c84220636a407606ec5291efd6770d7d45de5d048ab0123 WatchSource:0}: Error finding container 64db6bf74c50986c7c84220636a407606ec5291efd6770d7d45de5d048ab0123: Status 404 returned error can't find the container with id 64db6bf74c50986c7c84220636a407606ec5291efd6770d7d45de5d048ab0123
Sep 11 12:15:00.689356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.689322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:00.689531 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.689511    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:00.689601 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.689591    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls podName:8f3f287d-baa3-45f9-9108-429e7bf73b9a nodeName:}" failed. No retries permitted until 2026-09-11 12:15:01.689573496 +0000 UTC m=+37.080110839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls") pod "dns-default-p2rq7" (UID: "8f3f287d-baa3-45f9-9108-429e7bf73b9a") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:00.790210 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:00.790169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:00.790377 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.790288    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 12:15:00.790377 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.790299    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9bcfb56b-4z6m6: secret "image-registry-tls" not found
Sep 11 12:15:00.790377 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:00.790359    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls podName:3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:01.790345303 +0000 UTC m=+37.180882635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls") pod "image-registry-7d9bcfb56b-4z6m6" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee") : secret "image-registry-tls" not found
Sep 11 12:15:01.103408 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.103367    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:15:01.103408 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.103381    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:15:01.104162 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.103513    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:15:01.107160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.107135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhjsw\""
Sep 11 12:15:01.107266 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.107135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 12:15:01.107266 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.107135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 12:15:01.107266 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.107202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvbml\""
Sep 11 12:15:01.107762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.107747    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 12:15:01.108589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.108576    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 12:15:01.271327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.271299    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3ff7d73-8d6d-4152-98bc-cb8625f31cd7" containerID="6d268a7fa64be73edc10fcef94996aa8dab8a743b98d48846088a795c694ab90" exitCode=0
Sep 11 12:15:01.271570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.271381    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerDied","Data":"6d268a7fa64be73edc10fcef94996aa8dab8a743b98d48846088a795c694ab90"}
Sep 11 12:15:01.272899 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.272874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbh52" event={"ID":"2b5628a9-e957-4057-a70e-08d721e93717","Type":"ContainerStarted","Data":"64db6bf74c50986c7c84220636a407606ec5291efd6770d7d45de5d048ab0123"}
Sep 11 12:15:01.697456 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.697359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:01.697612 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:01.697531    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:01.697665 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:01.697612    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls podName:8f3f287d-baa3-45f9-9108-429e7bf73b9a nodeName:}" failed. No retries permitted until 2026-09-11 12:15:03.697589656 +0000 UTC m=+39.088126997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls") pod "dns-default-p2rq7" (UID: "8f3f287d-baa3-45f9-9108-429e7bf73b9a") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:01.798831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:01.798798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:01.799013 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:01.798969    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 12:15:01.799013 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:01.798992    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9bcfb56b-4z6m6: secret "image-registry-tls" not found
Sep 11 12:15:01.799131 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:01.799049    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls podName:3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:03.799035775 +0000 UTC m=+39.189573095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls") pod "image-registry-7d9bcfb56b-4z6m6" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee") : secret "image-registry-tls" not found
Sep 11 12:15:02.280197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:02.279942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" event={"ID":"f3ff7d73-8d6d-4152-98bc-cb8625f31cd7","Type":"ContainerStarted","Data":"460bad20429cf3ee6fab2c6c845bf3ae7ff90e0c9cfdd52b349687fb184c2c15"}
Sep 11 12:15:02.308674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:02.308615    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jlgq7" podStartSLOduration=6.2683615 podStartE2EDuration="36.308590471s" totalImagesPullingTime="30.040228971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:02.307655769 +0000 UTC m=+37.698193147" watchObservedRunningTime="2026-09-11 12:15:02.308590471 +0000 UTC m=+37.699127814"
Sep 11 12:15:02.312110 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:02.312070    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:02.410601 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:02.410566    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:02.522586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:02.522553    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:02.623932 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:02.623898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:03.715712 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:03.715676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:03.716056 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:03.715822    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:03.716056 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:03.715886    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls podName:8f3f287d-baa3-45f9-9108-429e7bf73b9a nodeName:}" failed. No retries permitted until 2026-09-11 12:15:07.715871877 +0000 UTC m=+43.106409200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls") pod "dns-default-p2rq7" (UID: "8f3f287d-baa3-45f9-9108-429e7bf73b9a") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:03.816733 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:03.816690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:03.816912 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:03.816875    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 12:15:03.816912 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:03.816891    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9bcfb56b-4z6m6: secret "image-registry-tls" not found
Sep 11 12:15:03.816978 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:03.816948    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls podName:3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:07.816930625 +0000 UTC m=+43.207467950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls") pod "image-registry-7d9bcfb56b-4z6m6" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee") : secret "image-registry-tls" not found
Sep 11 12:15:04.285985 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:04.285952    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbh52" event={"ID":"2b5628a9-e957-4057-a70e-08d721e93717","Type":"ContainerStarted","Data":"decb68c056525729df6c7e9083d629a2479b2165dd9f648d7ba7e2346998bb2c"}
Sep 11 12:15:04.302393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:04.302345    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rbh52" podStartSLOduration=2.67931079 podStartE2EDuration="5.302329027s" totalImagesPullingTime="2.623018237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:04.302003692 +0000 UTC m=+39.692541035" watchObservedRunningTime="2026-09-11 12:15:04.302329027 +0000 UTC m=+39.692866363"
Sep 11 12:15:05.518777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:05.518744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:06.296216 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:06.296182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:06.395057 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:06.395019    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:06.495719 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:06.495693    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:07.363455 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.363417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:07.464646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.464615    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:07.663249 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.663178    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2h56r"]
Sep 11 12:15:07.666812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.666783    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:07.694779 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.694750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2h56r"]
Sep 11 12:15:07.694938 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.694890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.698008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.697965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 12:15:07.698455 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.698342    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 12:15:07.698683 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.698660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 12:15:07.698784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.698762    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l5m57\""
Sep 11 12:15:07.698845 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.698820    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 12:15:07.734311 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.734284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:07.746331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.746307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:07.746427 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.746393    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:07.746474 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.746466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls podName:8f3f287d-baa3-45f9-9108-429e7bf73b9a nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.746453484 +0000 UTC m=+51.136990816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls") pod "dns-default-p2rq7" (UID: "8f3f287d-baa3-45f9-9108-429e7bf73b9a") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:07.847313 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.847278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-data-volume\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.847470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.847331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:07.847470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.847368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.847470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.847436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-crio-socket\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.847470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.847460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.847624 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.847491    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 12:15:07.847624 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.847509    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9bcfb56b-4z6m6: secret "image-registry-tls" not found
Sep 11 12:15:07.847624 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.847492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gsf\" (UniqueName: \"kubernetes.io/projected/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-api-access-97gsf\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.847624 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.847581    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls podName:3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.84756082 +0000 UTC m=+51.238098140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls") pod "image-registry-7d9bcfb56b-4z6m6" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee") : secret "image-registry-tls" not found
Sep 11 12:15:07.948181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-data-volume\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.948181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.948344 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-crio-socket\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.948344 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.948344 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gsf\" (UniqueName: \"kubernetes.io/projected/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-api-access-97gsf\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.948344 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.948293    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:07.948522 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.948346    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:07.948522 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.948373    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:08.448353082 +0000 UTC m=+43.838890433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:07.948522 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:07.948418    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:08.448383997 +0000 UTC m=+43.838921331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:07.948522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-crio-socket\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.948522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.948487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-data-volume\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:07.959136 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:07.959109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gsf\" (UniqueName: \"kubernetes.io/projected/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-api-access-97gsf\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:08.340842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:08.340811    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:08.441674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:08.441593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:08.452691 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:08.452669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:08.452803 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:08.452702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:08.452803 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:08.452779    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:08.452803 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:08.452797    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:08.452900 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:08.452847    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:09.452832886 +0000 UTC m=+44.843370220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:08.452900 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:08.452860    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:09.452854006 +0000 UTC m=+44.843391325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:08.667294 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:08.667186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:09.222127 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:09.222095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:09.312372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:09.312342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:09.459324 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:09.459292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:09.459324 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:09.459328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:09.459727 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:09.459425    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:09.459727 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:09.459467    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:09.459727 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:09.459486    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.459470296 +0000 UTC m=+46.850007630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:09.459727 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:09.459516    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.45950525 +0000 UTC m=+46.850042569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:09.824889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:09.824861    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhljr_63bae1b4-810c-4092-a4ba-cc816e956adc/dns-node-resolver/0.log"
Sep 11 12:15:10.544583 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.544553    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-br8rp"]
Sep 11 12:15:10.573370 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.573342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-br8rp"]
Sep 11 12:15:10.573534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.573446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.576145 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.576120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 12:15:10.576302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.576207    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 12:15:10.576371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.576316    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-djhtk\""
Sep 11 12:15:10.577019 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.577001    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 12:15:10.577107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.577030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 12:15:10.624703 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.624685    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6b29g_2dfc139d-214b-4e2f-8424-f97bc38788a0/node-ca/0.log"
Sep 11 12:15:10.668041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.668010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-signing-key\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.668143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.668081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-signing-cabundle\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.668143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.668101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92r2g\" (UniqueName: \"kubernetes.io/projected/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-kube-api-access-92r2g\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.768800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.768776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-signing-cabundle\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.768886 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.768808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r2g\" (UniqueName: \"kubernetes.io/projected/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-kube-api-access-92r2g\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.768886 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.768866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-signing-key\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.769534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.769518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-signing-cabundle\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.771128 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.771110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-signing-key\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.777159 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.777136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r2g\" (UniqueName: \"kubernetes.io/projected/8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c-kube-api-access-92r2g\") pod \"service-ca-568869fdc5-br8rp\" (UID: \"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c\") " pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:10.883013 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:10.882990    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-br8rp"
Sep 11 12:15:11.167176 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.167089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-br8rp"]
Sep 11 12:15:11.169784 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:11.169762    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbaf5fd_3aa9_4bd5_bf32_feb607ee7d7c.slice/crio-b658c6d13a5f22cd4262a537d0759f6d3cca4790f1fd0e017ce81aad4b2a67fd WatchSource:0}: Error finding container b658c6d13a5f22cd4262a537d0759f6d3cca4790f1fd0e017ce81aad4b2a67fd: Status 404 returned error can't find the container with id b658c6d13a5f22cd4262a537d0759f6d3cca4790f1fd0e017ce81aad4b2a67fd
Sep 11 12:15:11.302462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.302426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-br8rp" event={"ID":"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c","Type":"ContainerStarted","Data":"b658c6d13a5f22cd4262a537d0759f6d3cca4790f1fd0e017ce81aad4b2a67fd"}
Sep 11 12:15:11.344258 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.344223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:11.473893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.473821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:11.473893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.473862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:11.474058 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:11.473935    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:11.474058 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:11.473978    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:11.474058 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:11.473993    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.473979689 +0000 UTC m=+50.864517009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:11.474058 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:11.474020    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.474008092 +0000 UTC m=+50.864545428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:11.624069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.624041    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbh52_2b5628a9-e957-4057-a70e-08d721e93717/serve-healthcheck-canary/0.log"
Sep 11 12:15:11.915579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:11.915544    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:12.754873 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:12.754842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:14.310590 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:14.310551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-br8rp" event={"ID":"8cbaf5fd-3aa9-4bd5-bf32-feb607ee7d7c","Type":"ContainerStarted","Data":"90b2400e6e80ab4bf69d2f494684020da18d7c074d9a4069f11e162339ccb879"}
Sep 11 12:15:14.329672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:14.329610    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-br8rp" podStartSLOduration=1.872858714 podStartE2EDuration="4.329591078s" totalImagesPullingTime="2.456732364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:14.328933206 +0000 UTC m=+49.719470547" watchObservedRunningTime="2026-09-11 12:15:14.329591078 +0000 UTC m=+49.720128418"
Sep 11 12:15:14.605069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:14.605036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:15.506955 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.506914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:15.507428 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.506982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:15.507428 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.507151    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:15.507428 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.507229    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls podName:f56eb962-4c5c-4e68-a4c8-8cdf64e73d34 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:23.507208426 +0000 UTC m=+58.897745754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2h56r" (UID: "f56eb962-4c5c-4e68-a4c8-8cdf64e73d34") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:15.507602 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.507569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:15.614321 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.614275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:15.685167 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.685121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:15.809186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.809093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:15.809368 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.809327    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:15.809454 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.809417    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls podName:8f3f287d-baa3-45f9-9108-429e7bf73b9a nodeName:}" failed. No retries permitted until 2026-09-11 12:15:31.809383518 +0000 UTC m=+67.199920843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls") pod "dns-default-p2rq7" (UID: "8f3f287d-baa3-45f9-9108-429e7bf73b9a") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:15.910492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.910464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:15.910677 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.910651    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 12:15:15.910742 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.910678    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d9bcfb56b-4z6m6: secret "image-registry-tls" not found
Sep 11 12:15:15.910797 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:15.910746    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls podName:3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:31.910724089 +0000 UTC m=+67.301261417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls") pod "image-registry-7d9bcfb56b-4z6m6" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee") : secret "image-registry-tls" not found
Sep 11 12:15:15.963782 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:15.963741    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:16.067610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:16.067518    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:16.174848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:16.174746    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:16.264919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:16.264884    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:16.366013 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:16.365972    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:17.124171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:17.124140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:17.518854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:17.518771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:17.910072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:17.910045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:18.112837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:18.112789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:18.513641 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:18.513603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:18.581640 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:18.581600    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:19.015262 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:19.015224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:19.116314 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:19.116282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:19.449749 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:19.449697    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:19.709209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:19.709126    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:19.966033 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:19.965943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:20.066214 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:20.066179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:20.166872 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:20.166843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:20.644029 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:20.643989    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:21.310506 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:21.310477    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:21.715804 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:21.715768    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:21.837291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:21.837258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:21.939600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:21.939559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:22.114796 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:22.114744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:22.314925 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:22.314892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:23.127963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.127924    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:23.512779 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.512703    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:23.563774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.563740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:23.565879 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.565860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f56eb962-4c5c-4e68-a4c8-8cdf64e73d34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2h56r\" (UID: \"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34\") " pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:23.605869 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.605837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2h56r"
Sep 11 12:15:23.876698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.876661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2h56r"]
Sep 11 12:15:23.879283 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:23.879245    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56eb962_4c5c_4e68_a4c8_8cdf64e73d34.slice/crio-7d07abee2a8b59d7e5c45b740b24e22d08e7b37d3b7878f64af5b7bcae788f3a WatchSource:0}: Error finding container 7d07abee2a8b59d7e5c45b740b24e22d08e7b37d3b7878f64af5b7bcae788f3a: Status 404 returned error can't find the container with id 7d07abee2a8b59d7e5c45b740b24e22d08e7b37d3b7878f64af5b7bcae788f3a
Sep 11 12:15:23.917793 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:23.917767    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:24.117219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:24.117180    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:24.337601 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:24.337571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h56r" event={"ID":"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34","Type":"ContainerStarted","Data":"65f29f2fb69d6f83daaade38beb9e90a1d66843787cf2b79e79c20748b24c62d"}
Sep 11 12:15:24.337768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:24.337623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h56r" event={"ID":"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34","Type":"ContainerStarted","Data":"7d07abee2a8b59d7e5c45b740b24e22d08e7b37d3b7878f64af5b7bcae788f3a"}
Sep 11 12:15:24.526683 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:24.526646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:25.121995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:25.121911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:25.259279 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:25.259252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5szwd"
Sep 11 12:15:25.341978 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:25.341948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h56r" event={"ID":"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34","Type":"ContainerStarted","Data":"dfe2f671c83cdf2fc98e18c66d4d71b9644129883a3f07a25ab3df25f748f250"}
Sep 11 12:15:25.577259 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:25.577221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:25.915199 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:25.915120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:26.122084 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:26.122047    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:26.324520 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:26.324488    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:27.350276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:27.350240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2h56r" event={"ID":"f56eb962-4c5c-4e68-a4c8-8cdf64e73d34","Type":"ContainerStarted","Data":"17ffa01b3df52ac5a4dbe99e37fd689ae8c744b4b6b1ac0c8dcba38e59577089"}
Sep 11 12:15:27.375251 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:27.375201    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2h56r" podStartSLOduration=17.977463233 podStartE2EDuration="20.375186563s" totalImagesPullingTime="2.39772333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:15:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:27.3744179 +0000 UTC m=+62.764955236" watchObservedRunningTime="2026-09-11 12:15:27.375186563 +0000 UTC m=+62.765723915"
Sep 11 12:15:30.276951 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.276923    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"]
Sep 11 12:15:30.308123 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.308095    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.310020 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.309994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"]
Sep 11 12:15:30.310796 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.310775    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 12:15:30.311698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.311673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 12:15:30.311698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.311689    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 12:15:30.312001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.311673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 12:15:30.312001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.311726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 12:15:30.312001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.311790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 12:15:30.312110 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.312021    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 12:15:30.322805 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.322778    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:30.414259 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.414229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.414259 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.414260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68279eb-5f07-4ede-af3d-788427555ee4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.414450 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.414278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.414450 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.414386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-ca\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.414450 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.414419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-hub\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.414450 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.414444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nd9c\" (UniqueName: \"kubernetes.io/projected/a68279eb-5f07-4ede-af3d-788427555ee4-kube-api-access-7nd9c\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.418755 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.418735    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"]
Sep 11 12:15:30.418913 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:30.418888    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6" podUID="3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"
Sep 11 12:15:30.464215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.464179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:30.515625 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.515596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nd9c\" (UniqueName: \"kubernetes.io/projected/a68279eb-5f07-4ede-af3d-788427555ee4-kube-api-access-7nd9c\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.515757 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.515645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.515757 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.515665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68279eb-5f07-4ede-af3d-788427555ee4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.515876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.515771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.515876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.515856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-ca\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.515876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.515874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-hub\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.516452 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.516417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68279eb-5f07-4ede-af3d-788427555ee4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.518352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.518327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.518460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.518348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-hub\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.518460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.518443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-ca\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.518532 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.518492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a68279eb-5f07-4ede-af3d-788427555ee4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.550580 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.550527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nd9c\" (UniqueName: \"kubernetes.io/projected/a68279eb-5f07-4ede-af3d-788427555ee4-kube-api-access-7nd9c\") pod \"cluster-proxy-proxy-agent-6f849dbc54-dpdh5\" (UID: \"a68279eb-5f07-4ede-af3d-788427555ee4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.596125 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.596090    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:30.631855 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.631828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"
Sep 11 12:15:30.696821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.696785    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:30.792159 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.792127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:30.818464 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.818391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:15:30.820928 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.820910    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 12:15:30.830930 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.830912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7a698e7-ea0e-4c2b-afba-7c153c612e02-metrics-certs\") pod \"network-metrics-daemon-vb9f7\" (UID: \"e7a698e7-ea0e-4c2b-afba-7c153c612e02\") " pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:15:30.907471 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.907442    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:30.919462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.919435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:15:30.922395 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.922378    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 12:15:30.931891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.931866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87a282b2-6cba-47e6-8831-de744de9a5af-original-pull-secret\") pod \"global-pull-secret-syncer-2cbzp\" (UID: \"87a282b2-6cba-47e6-8831-de744de9a5af\") " pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:15:30.933006 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:30.932989    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5"]
Sep 11 12:15:30.935317 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:30.935292    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda68279eb_5f07_4ede_af3d_788427555ee4.slice/crio-03b6b017009dfaf565c01b2326f2edf900e10a26c6b12b496f8f9fb982050303 WatchSource:0}: Error finding container 03b6b017009dfaf565c01b2326f2edf900e10a26c6b12b496f8f9fb982050303: Status 404 returned error can't find the container with id 03b6b017009dfaf565c01b2326f2edf900e10a26c6b12b496f8f9fb982050303
Sep 11 12:15:31.020518 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.020494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:15:31.022776 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.022761    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 12:15:31.033190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.033170    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 12:15:31.043934 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.043915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qzm\" (UniqueName: \"kubernetes.io/projected/57232506-b657-42a9-b59b-df156dfae7f3-kube-api-access-g7qzm\") pod \"network-check-target-j4zck\" (UID: \"57232506-b657-42a9-b59b-df156dfae7f3\") " pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:15:31.084300 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.084261    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:31.116225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.116202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvbml\""
Sep 11 12:15:31.119268 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.119254    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2cbzp"
Sep 11 12:15:31.123971 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.123948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9f7"
Sep 11 12:15:31.129586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.129568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhjsw\""
Sep 11 12:15:31.134632 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.134612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:15:31.159378 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.159313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:31.361406 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.361320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5" event={"ID":"a68279eb-5f07-4ede-af3d-788427555ee4","Type":"ContainerStarted","Data":"03b6b017009dfaf565c01b2326f2edf900e10a26c6b12b496f8f9fb982050303"}
Sep 11 12:15:31.361406 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.361332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:31.366963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.366941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:31.424244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424218    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-installation-pull-secrets\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424261    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-image-registry-private-configuration\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424294    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-certificates\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-trusted-ca\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-ca-trust-extracted\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424584 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-bound-sa-token\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424584 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6tf5\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-kube-api-access-r6tf5\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:31.424707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:15:31.424759 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424737    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:15:31.424827 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.424804    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:15:31.426386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.426361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:15:31.426542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.426523    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:15:31.426542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.426532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:15:31.426629 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.426552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-kube-api-access-r6tf5" (OuterVolumeSpecName: "kube-api-access-r6tf5") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "kube-api-access-r6tf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:15:31.438143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.438118    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j4zck"]
Sep 11 12:15:31.438939 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.438921    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vb9f7"]
Sep 11 12:15:31.439748 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.439693    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2cbzp"]
Sep 11 12:15:31.439911 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:31.439892    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57232506_b657_42a9_b59b_df156dfae7f3.slice/crio-786141c1508833d875519e20292d2fbdce4b08f6e325b8d9da0e14523d7582bb WatchSource:0}: Error finding container 786141c1508833d875519e20292d2fbdce4b08f6e325b8d9da0e14523d7582bb: Status 404 returned error can't find the container with id 786141c1508833d875519e20292d2fbdce4b08f6e325b8d9da0e14523d7582bb
Sep 11 12:15:31.441029 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:31.441009    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a282b2_6cba_47e6_8831_de744de9a5af.slice/crio-d7a10f10cbc2e37b808f0a3f2afb9fa200cc316ba59742e900f088fdad3d1eeb WatchSource:0}: Error finding container d7a10f10cbc2e37b808f0a3f2afb9fa200cc316ba59742e900f088fdad3d1eeb: Status 404 returned error can't find the container with id d7a10f10cbc2e37b808f0a3f2afb9fa200cc316ba59742e900f088fdad3d1eeb
Sep 11 12:15:31.441828 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:31.441809    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a698e7_ea0e_4c2b_afba_7c153c612e02.slice/crio-5e718be6f51b411438d959f94f7c65a7eb3c3300963a332f2c0fba765ab75a99 WatchSource:0}: Error finding container 5e718be6f51b411438d959f94f7c65a7eb3c3300963a332f2c0fba765ab75a99: Status 404 returned error can't find the container with id 5e718be6f51b411438d959f94f7c65a7eb3c3300963a332f2c0fba765ab75a99
Sep 11 12:15:31.525569 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525538    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6tf5\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-kube-api-access-r6tf5\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.525569 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525565    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-installation-pull-secrets\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.525569 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525575    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-image-registry-private-configuration\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.525775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525584    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-certificates\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.525775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525593    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-trusted-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.525775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525602    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-ca-trust-extracted\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.525775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.525609    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-bound-sa-token\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:31.828610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.828487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:31.831317 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.831288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f3f287d-baa3-45f9-9108-429e7bf73b9a-metrics-tls\") pod \"dns-default-p2rq7\" (UID: \"8f3f287d-baa3-45f9-9108-429e7bf73b9a\") " pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:31.929182 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.929140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:31.932456 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:31.932428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"image-registry-7d9bcfb56b-4z6m6\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") " pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:32.029881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.029843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") pod \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\" (UID: \"3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee\") "
Sep 11 12:15:32.032868 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.032464    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" (UID: "3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:15:32.107443 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.107203    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgchp\""
Sep 11 12:15:32.115514 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.115482    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:32.130853 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.130830    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee-registry-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:15:32.366890 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.366779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2cbzp" event={"ID":"87a282b2-6cba-47e6-8831-de744de9a5af","Type":"ContainerStarted","Data":"d7a10f10cbc2e37b808f0a3f2afb9fa200cc316ba59742e900f088fdad3d1eeb"}
Sep 11 12:15:32.368705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.368630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb9f7" event={"ID":"e7a698e7-ea0e-4c2b-afba-7c153c612e02","Type":"ContainerStarted","Data":"5e718be6f51b411438d959f94f7c65a7eb3c3300963a332f2c0fba765ab75a99"}
Sep 11 12:15:32.371180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.371154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"
Sep 11 12:15:32.371711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.371685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4zck" event={"ID":"57232506-b657-42a9-b59b-df156dfae7f3","Type":"ContainerStarted","Data":"786141c1508833d875519e20292d2fbdce4b08f6e325b8d9da0e14523d7582bb"}
Sep 11 12:15:32.420599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.420549    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"]
Sep 11 12:15:32.423386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.423354    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d9bcfb56b-4z6m6"]
Sep 11 12:15:32.435646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:32.435500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2rq7"]
Sep 11 12:15:33.059870 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:33.059833    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3f287d_baa3_45f9_9108_429e7bf73b9a.slice/crio-89551f72a2ef26e95186444b97d5918f723cf03d0ba8ebab34d0f25a6924aa4a WatchSource:0}: Error finding container 89551f72a2ef26e95186444b97d5918f723cf03d0ba8ebab34d0f25a6924aa4a: Status 404 returned error can't find the container with id 89551f72a2ef26e95186444b97d5918f723cf03d0ba8ebab34d0f25a6924aa4a
Sep 11 12:15:33.106092 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:33.105959    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee" path="/var/lib/kubelet/pods/3d0a27c2-81fe-4bbd-926e-a2d9ce37d6ee/volumes"
Sep 11 12:15:33.378171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:33.378141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2rq7" event={"ID":"8f3f287d-baa3-45f9-9108-429e7bf73b9a","Type":"ContainerStarted","Data":"89551f72a2ef26e95186444b97d5918f723cf03d0ba8ebab34d0f25a6924aa4a"}
Sep 11 12:15:33.383892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:33.383864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:33.452441 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:33.452337    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:34.384727 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.384690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb9f7" event={"ID":"e7a698e7-ea0e-4c2b-afba-7c153c612e02","Type":"ContainerStarted","Data":"c46ee368d8b69e0d1269af22a75012f64a27ad844b0247684df3cd85e2187aec"}
Sep 11 12:15:34.385164 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.384758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb9f7" event={"ID":"e7a698e7-ea0e-4c2b-afba-7c153c612e02","Type":"ContainerStarted","Data":"73862395fb1bcb81a4216353e83010fa7137a04b9a27a922eae73057f0621895"}
Sep 11 12:15:34.477023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.476969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:34.570969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.570933    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:34.670107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.670010    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vb9f7" podStartSLOduration=66.751530108 podStartE2EDuration="1m8.66999374s" totalImagesPullingTime="1.918463632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:34.409737426 +0000 UTC m=+69.800274769" watchObservedRunningTime="2026-09-11 12:15:34.66999374 +0000 UTC m=+70.060531082"
Sep 11 12:15:34.672146 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.671021    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"]
Sep 11 12:15:34.672146 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.671592    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:34.673277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.673253    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:34.675953 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.675764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 12:15:34.675953 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.675857    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vj7hb\""
Sep 11 12:15:34.687457 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.687378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"]
Sep 11 12:15:34.736139 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.736012    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:34.754984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.754950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/811275af-c417-4c05-b143-ce812ea8d206-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9htxm\" (UID: \"811275af-c417-4c05-b143-ce812ea8d206\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:34.856033 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.855999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/811275af-c417-4c05-b143-ce812ea8d206-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9htxm\" (UID: \"811275af-c417-4c05-b143-ce812ea8d206\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:34.858984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.858958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/811275af-c417-4c05-b143-ce812ea8d206-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-9htxm\" (UID: \"811275af-c417-4c05-b143-ce812ea8d206\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:34.903417 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.903368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:34.984890 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:34.984813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:37.849157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:37.849079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"]
Sep 11 12:15:38.044249 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:38.044211    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811275af_c417_4c05_b143_ce812ea8d206.slice/crio-2eb024a75ee1b909b654dbd97b599a94ad7c09978c081b293e2bc395ec5eabf3 WatchSource:0}: Error finding container 2eb024a75ee1b909b654dbd97b599a94ad7c09978c081b293e2bc395ec5eabf3: Status 404 returned error can't find the container with id 2eb024a75ee1b909b654dbd97b599a94ad7c09978c081b293e2bc395ec5eabf3
Sep 11 12:15:38.405162 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.405085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2cbzp" event={"ID":"87a282b2-6cba-47e6-8831-de744de9a5af","Type":"ContainerStarted","Data":"37f36059e9f4920c3a117705379e8ab9017b4363f1c25466a945d1687161e7f2"}
Sep 11 12:15:38.406942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.406913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2rq7" event={"ID":"8f3f287d-baa3-45f9-9108-429e7bf73b9a","Type":"ContainerStarted","Data":"55c5a049d66cebd595ed63f052de4e213147c165592dc3bd05087f9f5e70dc2d"}
Sep 11 12:15:38.407236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.406953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2rq7" event={"ID":"8f3f287d-baa3-45f9-9108-429e7bf73b9a","Type":"ContainerStarted","Data":"403c2e77a6a7138c6904d80d6e93b8ac7308549ccb347cd9698ee05c3f90b96d"}
Sep 11 12:15:38.407236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.406974    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:38.408145 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.408123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm" event={"ID":"811275af-c417-4c05-b143-ce812ea8d206","Type":"ContainerStarted","Data":"2eb024a75ee1b909b654dbd97b599a94ad7c09978c081b293e2bc395ec5eabf3"}
Sep 11 12:15:38.410166 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.410145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5" event={"ID":"a68279eb-5f07-4ede-af3d-788427555ee4","Type":"ContainerStarted","Data":"c95a47a5c64d8f1f8e160fccd22e4857a9e0a2c849732f956052538361fcbc7d"}
Sep 11 12:15:38.410292 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.410176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5" event={"ID":"a68279eb-5f07-4ede-af3d-788427555ee4","Type":"ContainerStarted","Data":"f7677a39a0d3c2e5c46557fbbcdd8f1b51ea89f9c1e9f4047ac59464f4e69cea"}
Sep 11 12:15:38.410292 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.410189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5" event={"ID":"a68279eb-5f07-4ede-af3d-788427555ee4","Type":"ContainerStarted","Data":"a2fd7515e39969bd7da6bea3710639394c2aa9b3440540db72623c897b8ce90e"}
Sep 11 12:15:38.411500 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.411481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j4zck" event={"ID":"57232506-b657-42a9-b59b-df156dfae7f3","Type":"ContainerStarted","Data":"58ea86cb1f2c9a05b69312abba3a7ac61bd7013de5731d238f32b76d67313129"}
Sep 11 12:15:38.411615 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.411603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:15:38.426270 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.425293    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2cbzp" podStartSLOduration=65.823767608 podStartE2EDuration="1m12.425270777s" totalImagesPullingTime="6.601503169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:38.424360002 +0000 UTC m=+73.814897346" watchObservedRunningTime="2026-09-11 12:15:38.425270777 +0000 UTC m=+73.815808120"
Sep 11 12:15:38.447340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.447298    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f849dbc54-dpdh5" podStartSLOduration=1.800218866 podStartE2EDuration="8.44728628s" totalImagesPullingTime="6.647067414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:38.44572699 +0000 UTC m=+73.836264331" watchObservedRunningTime="2026-09-11 12:15:38.44728628 +0000 UTC m=+73.837823619"
Sep 11 12:15:38.469147 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.468832    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p2rq7" podStartSLOduration=34.434449554 podStartE2EDuration="39.468816899s" totalImagesPullingTime="5.034367345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:38.467278434 +0000 UTC m=+73.857815776" watchObservedRunningTime="2026-09-11 12:15:38.468816899 +0000 UTC m=+73.859354245"
Sep 11 12:15:38.489924 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:38.489776    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j4zck" podStartSLOduration=65.887676579 podStartE2EDuration="1m12.489759689s" totalImagesPullingTime="6.60208311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:38.488629624 +0000 UTC m=+73.879166966" watchObservedRunningTime="2026-09-11 12:15:38.489759689 +0000 UTC m=+73.880297034"
Sep 11 12:15:40.421157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.421121    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm" event={"ID":"811275af-c417-4c05-b143-ce812ea8d206","Type":"ContainerStarted","Data":"6c4ad11b734c017a8b5aca9728177beffd4aee91e050b9139e422b228353e6ad"}
Sep 11 12:15:40.421557 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.421308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:40.426036 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.426015    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm"
Sep 11 12:15:40.441351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.441312    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-9htxm" podStartSLOduration=4.983215719 podStartE2EDuration="6.441299426s" totalImagesPullingTime="1.458083707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:40.439635431 +0000 UTC m=+75.830172774" watchObservedRunningTime="2026-09-11 12:15:40.441299426 +0000 UTC m=+75.831836768"
Sep 11 12:15:40.723549 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.723475    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:40.801878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.801833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:40.910236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:40.910199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:41.424616 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:41.424578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:41.524872 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:41.524839    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:41.629636 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:41.629607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:43.441606 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:43.441568    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:44.929185 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:44.929154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:45.049881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.049846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:45.203803 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.203732    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"]
Sep 11 12:15:45.207601 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.207566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.210282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.210258    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 12:15:45.210618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.210596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 12:15:45.211431 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.211389    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 12:15:45.211527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.211454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bgt62\""
Sep 11 12:15:45.211527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.211515    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 12:15:45.211656 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.211640    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 12:15:45.220966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.220483    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"]
Sep 11 12:15:45.251022 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.250996    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4mqfw"]
Sep 11 12:15:45.254027 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.253910    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.256565 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.256439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 12:15:45.256959 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.256944    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 12:15:45.257108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.257093    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bv4vc\""
Sep 11 12:15:45.258680 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.258654    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 12:15:45.326115 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a0ba35d-63b1-46b1-adae-b265d3534b9c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.326115 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldkb\" (UniqueName: \"kubernetes.io/projected/4a0ba35d-63b1-46b1-adae-b265d3534b9c-kube-api-access-dldkb\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.326339 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-root\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326339 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9115c913-bc58-4ded-a06f-53c8c67feb43-metrics-client-ca\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326339 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-accelerators-collector-config\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326339 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a0ba35d-63b1-46b1-adae-b265d3534b9c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.326524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-tls\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-wtmp\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-textfile\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ba35d-63b1-46b1-adae-b265d3534b9c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.326524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvw9f\" (UniqueName: \"kubernetes.io/projected/9115c913-bc58-4ded-a06f-53c8c67feb43-kube-api-access-mvw9f\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326685 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-sys\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.326685 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.326593    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.396233 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.396199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:45.427541 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a0ba35d-63b1-46b1-adae-b265d3534b9c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.427541 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldkb\" (UniqueName: \"kubernetes.io/projected/4a0ba35d-63b1-46b1-adae-b265d3534b9c-kube-api-access-dldkb\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-root\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9115c913-bc58-4ded-a06f-53c8c67feb43-metrics-client-ca\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-accelerators-collector-config\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a0ba35d-63b1-46b1-adae-b265d3534b9c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-tls\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-wtmp\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.427737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-textfile\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-root\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ba35d-63b1-46b1-adae-b265d3534b9c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvw9f\" (UniqueName: \"kubernetes.io/projected/9115c913-bc58-4ded-a06f-53c8c67feb43-kube-api-access-mvw9f\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.427837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-sys\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:45.427915    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:15:45.427970    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-tls podName:9115c913-bc58-4ded-a06f-53c8c67feb43 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:45.927951967 +0000 UTC m=+81.318489302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-tls") pod "node-exporter-4mqfw" (UID: "9115c913-bc58-4ded-a06f-53c8c67feb43") : secret "node-exporter-tls" not found
Sep 11 12:15:45.428089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-sys\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-wtmp\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9115c913-bc58-4ded-a06f-53c8c67feb43-metrics-client-ca\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a0ba35d-63b1-46b1-adae-b265d3534b9c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.428681 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-textfile\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.428734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.428691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-accelerators-collector-config\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.430393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.430372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.430651 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.430629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a0ba35d-63b1-46b1-adae-b265d3534b9c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.430829 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.430810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a0ba35d-63b1-46b1-adae-b265d3534b9c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.441082 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.441062    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvw9f\" (UniqueName: \"kubernetes.io/projected/9115c913-bc58-4ded-a06f-53c8c67feb43-kube-api-access-mvw9f\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.441201 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.441136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldkb\" (UniqueName: \"kubernetes.io/projected/4a0ba35d-63b1-46b1-adae-b265d3534b9c-kube-api-access-dldkb\") pod \"openshift-state-metrics-67dcf8c979-mqm5b\" (UID: \"4a0ba35d-63b1-46b1-adae-b265d3534b9c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.519845 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.519780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"
Sep 11 12:15:45.718371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.718319    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:45.799897 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.799833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b"]
Sep 11 12:15:45.802763 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:45.802735    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0ba35d_63b1_46b1_adae_b265d3534b9c.slice/crio-49af68ff3db6e23f534f8e0c3eda4f8ec9b08b6f4645ab1d124d6eaa7b867b09 WatchSource:0}: Error finding container 49af68ff3db6e23f534f8e0c3eda4f8ec9b08b6f4645ab1d124d6eaa7b867b09: Status 404 returned error can't find the container with id 49af68ff3db6e23f534f8e0c3eda4f8ec9b08b6f4645ab1d124d6eaa7b867b09
Sep 11 12:15:45.931919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.931890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-tls\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:45.933984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:45.933964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9115c913-bc58-4ded-a06f-53c8c67feb43-node-exporter-tls\") pod \"node-exporter-4mqfw\" (UID: \"9115c913-bc58-4ded-a06f-53c8c67feb43\") " pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:46.055744 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.055669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:46.163543 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.163507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4mqfw"
Sep 11 12:15:46.167848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.167819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:46.173638 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:46.173617    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9115c913_bc58_4ded_a06f_53c8c67feb43.slice/crio-f28f348f94e1222e99821eae968ab07143258f4da34fba25168665cc94a02488 WatchSource:0}: Error finding container f28f348f94e1222e99821eae968ab07143258f4da34fba25168665cc94a02488: Status 404 returned error can't find the container with id f28f348f94e1222e99821eae968ab07143258f4da34fba25168665cc94a02488
Sep 11 12:15:46.286171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.286137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:46.418680 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.418640    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:15:46.422361 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.422344    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.425279 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425251    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 12:15:46.425436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 12:15:46.425436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425371    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 12:15:46.425436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 12:15:46.425586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 12:15:46.425808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425789    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2j67m\""
Sep 11 12:15:46.425952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425841    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 12:15:46.426028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.425959    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 12:15:46.426093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.426077    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 12:15:46.426140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.426092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 12:15:46.442163 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.442144    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:15:46.445825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.445797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4mqfw" event={"ID":"9115c913-bc58-4ded-a06f-53c8c67feb43","Type":"ContainerStarted","Data":"f28f348f94e1222e99821eae968ab07143258f4da34fba25168665cc94a02488"}
Sep 11 12:15:46.447868 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.447842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b" event={"ID":"4a0ba35d-63b1-46b1-adae-b265d3534b9c","Type":"ContainerStarted","Data":"fa003ce00414fc1c08a1ea08987109826d817e8b97966fda86768a7344571a2f"}
Sep 11 12:15:46.447944 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.447897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b" event={"ID":"4a0ba35d-63b1-46b1-adae-b265d3534b9c","Type":"ContainerStarted","Data":"a6eb9a5bcd4a8c02a9a1e7385e40dd7dc9b262743fbd69f50aef1e5e2049e071"}
Sep 11 12:15:46.447944 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.447927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b" event={"ID":"4a0ba35d-63b1-46b1-adae-b265d3534b9c","Type":"ContainerStarted","Data":"49af68ff3db6e23f534f8e0c3eda4f8ec9b08b6f4645ab1d124d6eaa7b867b09"}
Sep 11 12:15:46.536435 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccc8q\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-kube-api-access-ccc8q\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-out\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-web-config\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.536825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.536770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-volume\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.601637 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.601607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:46.637588 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.637751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.637820 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.637936 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-volume\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.637936 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccc8q\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-kube-api-access-ccc8q\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.637936 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.637986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.638007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-out\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.638048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638343 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.638094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638343 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.638122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.638343 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.638164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-web-config\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.639179 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.639151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.639380 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.639357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.641336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.641055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-web-config\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.642104 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.642035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.642198 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.642176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.642360 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.642335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.642628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.642605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-out\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.643341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.643314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.643476 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.643388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-volume\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.643770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.643747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.644679 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.644658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.647160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.647137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccc8q\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-kube-api-access-ccc8q\") pod \"alertmanager-main-0\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:46.698066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.697991    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:46.732831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:46.732804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:15:47.033433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:47.033271    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:15:47.063266 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:47.063236    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13038ad5_116c_4a19_91a0_ad2d2ece1600.slice/crio-ae3c07ffb14a6555354de8831d4924a1131d809992a88a7ebc8bbe8f6938a8ca WatchSource:0}: Error finding container ae3c07ffb14a6555354de8831d4924a1131d809992a88a7ebc8bbe8f6938a8ca: Status 404 returned error can't find the container with id ae3c07ffb14a6555354de8831d4924a1131d809992a88a7ebc8bbe8f6938a8ca
Sep 11 12:15:47.453478 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:47.453443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4mqfw" event={"ID":"9115c913-bc58-4ded-a06f-53c8c67feb43","Type":"ContainerStarted","Data":"8db7d3aed7c17aef165584a2064b7643e52abef5d88d65b209187206598c278b"}
Sep 11 12:15:47.454933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:47.454907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"ae3c07ffb14a6555354de8831d4924a1131d809992a88a7ebc8bbe8f6938a8ca"}
Sep 11 12:15:47.457196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:47.457172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b" event={"ID":"4a0ba35d-63b1-46b1-adae-b265d3534b9c","Type":"ContainerStarted","Data":"396a7930d936084949798469024d4f765d5b9cbe651e4c7e1fe55bf59b455ead"}
Sep 11 12:15:47.528360 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:47.528302    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqm5b" podStartSLOduration=1.11967251 podStartE2EDuration="2.528284684s" totalImagesPullingTime="1.408612174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:47.526946049 +0000 UTC m=+82.917483388" watchObservedRunningTime="2026-09-11 12:15:47.528284684 +0000 UTC m=+82.918822025"
Sep 11 12:15:48.224096 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.224058    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:48.347746 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.347704    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:48.417802 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.417764    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p2rq7"
Sep 11 12:15:48.430780 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.430747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:48.463030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.462997    2561 generic.go:356] "Generic (PLEG): container finished" podID="9115c913-bc58-4ded-a06f-53c8c67feb43" containerID="8db7d3aed7c17aef165584a2064b7643e52abef5d88d65b209187206598c278b" exitCode=0
Sep 11 12:15:48.463202 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.463039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4mqfw" event={"ID":"9115c913-bc58-4ded-a06f-53c8c67feb43","Type":"ContainerDied","Data":"8db7d3aed7c17aef165584a2064b7643e52abef5d88d65b209187206598c278b"}
Sep 11 12:15:48.600784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:48.600752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:49.469002 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:49.468960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4mqfw" event={"ID":"9115c913-bc58-4ded-a06f-53c8c67feb43","Type":"ContainerStarted","Data":"a8f99435a2f87019a25c6e8f2ce6656ee7cfb1c354ad7a7995f512abefc224cc"}
Sep 11 12:15:49.469495 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:49.469030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4mqfw" event={"ID":"9115c913-bc58-4ded-a06f-53c8c67feb43","Type":"ContainerStarted","Data":"ab14b2ebcea01cb414bd09d460e2a686be7f491e3dae199efd39d03c8df5e7da"}
Sep 11 12:15:49.470547 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:49.470525    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f" exitCode=0
Sep 11 12:15:49.470652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:49.470577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"}
Sep 11 12:15:49.524054 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:49.524008    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4mqfw" podStartSLOduration=3.316854767 podStartE2EDuration="4.523994027s" totalImagesPullingTime="1.144226165s" totalInitContainerRuntime="62.913095ms" isStatefulPod=true podCreationTimestamp="2026-09-11 12:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:49.492450278 +0000 UTC m=+84.882987619" watchObservedRunningTime="2026-09-11 12:15:49.523994027 +0000 UTC m=+84.914531369"
Sep 11 12:15:49.846382 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:49.846336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.005493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.005461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.065695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.065651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.166568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.166480    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.426103 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.426018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.526106 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.526062    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.676874 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.676789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.753440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.753393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.829612 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.829580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:50.930717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:50.930636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:51.245254 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.245169    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:51.450592 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.450562    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:51.479493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.479464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"}
Sep 11 12:15:51.479584 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.479547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"}
Sep 11 12:15:51.614063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.614038    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:15:51.618135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.618115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.620899 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.620853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:51.621387 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.621350    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 12:15:51.622930 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.622911    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 12:15:51.624074 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.624050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 12:15:51.625190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.624788    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 12:15:51.625276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.625256    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 12:15:51.625545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.625525    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 12:15:51.626118 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.626100    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 12:15:51.626184 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.626142    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-px227\""
Sep 11 12:15:51.626493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.626468    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 12:15:51.626587 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.626540    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 12:15:51.626733 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.626717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 12:15:51.626964 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.626943    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bloc51pfupm6d\""
Sep 11 12:15:51.627072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.627006    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 12:15:51.627072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.627040    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 12:15:51.627643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.627443    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 12:15:51.640351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.640334    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 12:15:51.648489 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.648471    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 12:15:51.653567 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.653546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:15:51.678961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.678942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.678971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.678993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-config\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679147 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679147 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679147 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-kube-api-access-2w9rq\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.679597 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.679489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.780849 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.780849 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.780849 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.780849 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-kube-api-access-2w9rq\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.780995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.781017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781552 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.781526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.781614 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.781592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782118 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782118 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782118 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782181    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.782291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.783848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.782977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-config\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.783848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.783015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.783848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.783052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.783848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.783085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.784518 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.784497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.784974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.785053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.785226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.785423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.785509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.785795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.785942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.785920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.786358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.786327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-config\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.786647 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.786618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.786904 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.786882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.787186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.787165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.787557 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.787540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.787597 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.787566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.791367 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.791350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-kube-api-access-2w9rq\") pod \"prometheus-k8s-0\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:51.856673 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.856642    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:51.928536 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:51.928506    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:15:52.217638 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.217614    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:15:52.219602 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:15:52.219574    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9022825_1fa0_4d79_9784_9e5fc624a245.slice/crio-ee281b50da3d8463d652aaf875bcc87db2995003c6bd19d3522552739769b7aa WatchSource:0}: Error finding container ee281b50da3d8463d652aaf875bcc87db2995003c6bd19d3522552739769b7aa: Status 404 returned error can't find the container with id ee281b50da3d8463d652aaf875bcc87db2995003c6bd19d3522552739769b7aa
Sep 11 12:15:52.262319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.262270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:52.367235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.367193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:52.463903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.463826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:52.488888 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.488817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"}
Sep 11 12:15:52.488888 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.488887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"}
Sep 11 12:15:52.489075 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.488926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"}
Sep 11 12:15:52.490581 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.490549    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8" exitCode=0
Sep 11 12:15:52.490691 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.490633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"}
Sep 11 12:15:52.490782 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.490709    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"ee281b50da3d8463d652aaf875bcc87db2995003c6bd19d3522552739769b7aa"}
Sep 11 12:15:52.539225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:52.539173    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:53.499572 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:53.499541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerStarted","Data":"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"}
Sep 11 12:15:53.529709 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:53.529649    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.739955633 podStartE2EDuration="7.529630539s" totalImagesPullingTime="4.77112746s" totalInitContainerRuntime="18.547446ms" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:53.528266425 +0000 UTC m=+88.918803790" watchObservedRunningTime="2026-09-11 12:15:53.529630539 +0000 UTC m=+88.920167882"
Sep 11 12:15:53.675539 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:53.675498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:53.778877 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:53.778793    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:56.515542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:56.515495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"}
Sep 11 12:15:56.516025 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:56.515585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"}
Sep 11 12:15:57.124516 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:57.124485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:57.224898 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:57.224858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:58.526914 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.526883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"}
Sep 11 12:15:58.527282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.526945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"}
Sep 11 12:15:58.527282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.526970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"}
Sep 11 12:15:58.527282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.526997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerStarted","Data":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"}
Sep 11 12:15:58.563462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.563412    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.275028185 podStartE2EDuration="7.563379981s" totalImagesPullingTime="5.266761152s" totalInitContainerRuntime="21.590644ms" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:58.561277188 +0000 UTC m=+93.951814562" watchObservedRunningTime="2026-09-11 12:15:58.563379981 +0000 UTC m=+93.953917322"
Sep 11 12:15:58.663758 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.663726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:15:58.726526 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:15:58.726494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:00.509067 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:00.509024    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:00.613424 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:00.613376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:01.234906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:01.234861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:01.350120 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:01.350086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:01.929104 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:01.929060    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:16:02.539984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:02.539942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:04.072631 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:04.072596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:04.172947 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:04.172915    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:06.232051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:06.232003    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:06.302658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:06.302617    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:07.489921 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:07.489881    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:07.579671 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:07.579603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:07.717663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:07.717628    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:07.880382 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:07.880349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:07.966008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:07.965973    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:09.141292 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:09.141256    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:09.232688 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:09.232657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:09.418219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:09.418138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j4zck"
Sep 11 12:16:11.675005 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:11.674970    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:15.307592 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:15.307546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:15.407823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:15.407789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:16.868593 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:16.868554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:16.927466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:16.927437    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:18.746980 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:18.746939    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:18.847358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:18.847325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:19.096882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:19.096848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:30.269499 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:30.269451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:30.426563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:30.426525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:16:51.928772 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:51.928742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:16:51.947240 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:51.947216    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:16:52.739513 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:16:52.739485    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:04.675041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:04.674992    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:04.875238 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:04.875199    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:05.777787 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.777746    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:17:05.778165 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.778029    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="alertmanager" containerID="cri-o://961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84" gracePeriod=120
Sep 11 12:17:05.778165 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.778090    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-metric" containerID="cri-o://b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594" gracePeriod=120
Sep 11 12:17:05.778165 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.778127    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-web" containerID="cri-o://2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39" gracePeriod=120
Sep 11 12:17:05.778319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.778166    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="prom-label-proxy" containerID="cri-o://ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1" gracePeriod=120
Sep 11 12:17:05.778319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.778175    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="config-reloader" containerID="cri-o://ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6" gracePeriod=120
Sep 11 12:17:05.778319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:05.778201    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy" containerID="cri-o://c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504" gracePeriod=120
Sep 11 12:17:06.776842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776812    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1" exitCode=0
Sep 11 12:17:06.776842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776838    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504" exitCode=0
Sep 11 12:17:06.776842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776847    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6" exitCode=0
Sep 11 12:17:06.776842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776853    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84" exitCode=0
Sep 11 12:17:06.777108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"}
Sep 11 12:17:06.777108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"}
Sep 11 12:17:06.777108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.776980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"}
Sep 11 12:17:06.777108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:06.777004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"}
Sep 11 12:17:07.024643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.024617    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:07.173379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173340    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-metric\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173379    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-web-config\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173591 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173442    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-trusted-ca-bundle\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173591 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-cluster-tls-config\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173591 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-main-tls\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173591 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173548    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-main-db\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173591 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccc8q\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-kube-api-access-ccc8q\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173802 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-metrics-client-ca\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173802 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-out\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.173802 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.173652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-web\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.174049 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.174027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.174117 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.174093    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-volume\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.174172 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.174114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:07.174172 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.174141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-tls-assets\") pod \"13038ad5-116c-4a19-91a0-ad2d2ece1600\" (UID: \"13038ad5-116c-4a19-91a0-ad2d2ece1600\") "
Sep 11 12:17:07.174698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.174314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:07.174698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.174588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:07.177305 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.177279    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.177449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.177311    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-alertmanager-main-db\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.177449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.177327    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13038ad5-116c-4a19-91a0-ad2d2ece1600-metrics-client-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.177449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.177428    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.178051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.178020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:17:07.178644 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.178606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.178733 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.178652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-kube-api-access-ccc8q" (OuterVolumeSpecName: "kube-api-access-ccc8q") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "kube-api-access-ccc8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:17:07.179523 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.179497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.179762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.179741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.179832 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.179782    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-volume" (OuterVolumeSpecName: "config-volume") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.182882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.182598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-out" (OuterVolumeSpecName: "config-out") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:07.184440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.184417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.192879 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.192857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-web-config" (OuterVolumeSpecName: "web-config") pod "13038ad5-116c-4a19-91a0-ad2d2ece1600" (UID: "13038ad5-116c-4a19-91a0-ad2d2ece1600"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:07.278081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278055    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278079    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-web-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278088    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-cluster-tls-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278097    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-main-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278105    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccc8q\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-kube-api-access-ccc8q\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278113    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-out\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278121    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278130    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278139    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13038ad5-116c-4a19-91a0-ad2d2ece1600-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.278215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.278146    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13038ad5-116c-4a19-91a0-ad2d2ece1600-tls-assets\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:07.783568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783531    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594" exitCode=0
Sep 11 12:17:07.783568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783564    2561 generic.go:356] "Generic (PLEG): container finished" podID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerID="2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39" exitCode=0
Sep 11 12:17:07.783753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"}
Sep 11 12:17:07.783753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783642    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:07.783753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"}
Sep 11 12:17:07.783753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13038ad5-116c-4a19-91a0-ad2d2ece1600","Type":"ContainerDied","Data":"ae3c07ffb14a6555354de8831d4924a1131d809992a88a7ebc8bbe8f6938a8ca"}
Sep 11 12:17:07.783879 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.783755    2561 scope.go:117] "RemoveContainer" containerID="ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"
Sep 11 12:17:07.791461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.791446    2561 scope.go:117] "RemoveContainer" containerID="b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"
Sep 11 12:17:07.798672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.798654    2561 scope.go:117] "RemoveContainer" containerID="c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"
Sep 11 12:17:07.805019 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.804999    2561 scope.go:117] "RemoveContainer" containerID="2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"
Sep 11 12:17:07.811506 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.811474    2561 scope.go:117] "RemoveContainer" containerID="ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"
Sep 11 12:17:07.817915 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.817899    2561 scope.go:117] "RemoveContainer" containerID="961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"
Sep 11 12:17:07.824305 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.824287    2561 scope.go:117] "RemoveContainer" containerID="6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"
Sep 11 12:17:07.830632 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.830617    2561 scope.go:117] "RemoveContainer" containerID="ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"
Sep 11 12:17:07.830866 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.830845    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1\": container with ID starting with ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1 not found: ID does not exist" containerID="ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"
Sep 11 12:17:07.830936 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.830877    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"} err="failed to get container status \"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1\": rpc error: code = NotFound desc = could not find container \"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1\": container with ID starting with ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1 not found: ID does not exist"
Sep 11 12:17:07.830936 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.830917    2561 scope.go:117] "RemoveContainer" containerID="b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"
Sep 11 12:17:07.831166 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.831147    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594\": container with ID starting with b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594 not found: ID does not exist" containerID="b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"
Sep 11 12:17:07.831211 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831172    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"} err="failed to get container status \"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594\": rpc error: code = NotFound desc = could not find container \"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594\": container with ID starting with b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594 not found: ID does not exist"
Sep 11 12:17:07.831211 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831188    2561 scope.go:117] "RemoveContainer" containerID="c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"
Sep 11 12:17:07.831441 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.831422    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504\": container with ID starting with c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504 not found: ID does not exist" containerID="c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"
Sep 11 12:17:07.831511 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831448    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"} err="failed to get container status \"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504\": rpc error: code = NotFound desc = could not find container \"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504\": container with ID starting with c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504 not found: ID does not exist"
Sep 11 12:17:07.831511 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831469    2561 scope.go:117] "RemoveContainer" containerID="2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"
Sep 11 12:17:07.831700 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.831684    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39\": container with ID starting with 2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39 not found: ID does not exist" containerID="2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"
Sep 11 12:17:07.831736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831704    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"} err="failed to get container status \"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39\": rpc error: code = NotFound desc = could not find container \"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39\": container with ID starting with 2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39 not found: ID does not exist"
Sep 11 12:17:07.831736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831719    2561 scope.go:117] "RemoveContainer" containerID="ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"
Sep 11 12:17:07.831940 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.831920    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6\": container with ID starting with ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6 not found: ID does not exist" containerID="ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"
Sep 11 12:17:07.831982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831946    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"} err="failed to get container status \"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6\": rpc error: code = NotFound desc = could not find container \"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6\": container with ID starting with ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6 not found: ID does not exist"
Sep 11 12:17:07.831982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.831961    2561 scope.go:117] "RemoveContainer" containerID="961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"
Sep 11 12:17:07.832182 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.832167    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84\": container with ID starting with 961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84 not found: ID does not exist" containerID="961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"
Sep 11 12:17:07.832235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832189    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"} err="failed to get container status \"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84\": rpc error: code = NotFound desc = could not find container \"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84\": container with ID starting with 961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84 not found: ID does not exist"
Sep 11 12:17:07.832235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832208    2561 scope.go:117] "RemoveContainer" containerID="6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"
Sep 11 12:17:07.832436 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:07.832420    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f\": container with ID starting with 6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f not found: ID does not exist" containerID="6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"
Sep 11 12:17:07.832489 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832438    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"} err="failed to get container status \"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f\": rpc error: code = NotFound desc = could not find container \"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f\": container with ID starting with 6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f not found: ID does not exist"
Sep 11 12:17:07.832489 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832450    2561 scope.go:117] "RemoveContainer" containerID="ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"
Sep 11 12:17:07.832672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832655    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1"} err="failed to get container status \"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1\": rpc error: code = NotFound desc = could not find container \"ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1\": container with ID starting with ddbfdaf9a4d0298ebe71a94e8374d2e07893b79c889708e21a0c55c9647987d1 not found: ID does not exist"
Sep 11 12:17:07.832733 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832674    2561 scope.go:117] "RemoveContainer" containerID="b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"
Sep 11 12:17:07.832883 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832864    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594"} err="failed to get container status \"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594\": rpc error: code = NotFound desc = could not find container \"b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594\": container with ID starting with b74c39c93080969671244fca4991dfc2e3188f1577ac45d0307c6525aac52594 not found: ID does not exist"
Sep 11 12:17:07.832927 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.832884    2561 scope.go:117] "RemoveContainer" containerID="c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"
Sep 11 12:17:07.833090 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833075    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504"} err="failed to get container status \"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504\": rpc error: code = NotFound desc = could not find container \"c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504\": container with ID starting with c371340dc1172eb70f6e209d45c504720af34389fe57cfef1a838ad84513b504 not found: ID does not exist"
Sep 11 12:17:07.833153 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833091    2561 scope.go:117] "RemoveContainer" containerID="2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"
Sep 11 12:17:07.833290 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833275    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39"} err="failed to get container status \"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39\": rpc error: code = NotFound desc = could not find container \"2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39\": container with ID starting with 2029f426064d4dc7a80692f1ef9fe0b28b4da81511b874913feff7bfa6687e39 not found: ID does not exist"
Sep 11 12:17:07.833333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833290    2561 scope.go:117] "RemoveContainer" containerID="ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"
Sep 11 12:17:07.833492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833473    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6"} err="failed to get container status \"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6\": rpc error: code = NotFound desc = could not find container \"ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6\": container with ID starting with ff3445cb9d8971d25a048a36fd629739e46c3c8f02dff9658ef0117288b771f6 not found: ID does not exist"
Sep 11 12:17:07.833566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833492    2561 scope.go:117] "RemoveContainer" containerID="961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"
Sep 11 12:17:07.833702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833687    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84"} err="failed to get container status \"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84\": rpc error: code = NotFound desc = could not find container \"961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84\": container with ID starting with 961713d73e99252a8fb51aa3db0f45a3399deeeb37af7b88f4ea558524c73b84 not found: ID does not exist"
Sep 11 12:17:07.833767 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833703    2561 scope.go:117] "RemoveContainer" containerID="6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"
Sep 11 12:17:07.833937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.833921    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f"} err="failed to get container status \"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f\": rpc error: code = NotFound desc = could not find container \"6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f\": container with ID starting with 6df7eed5fb01d83a5cdb4c7b5a3ec674f69b4bdedbaa6caeba485068259e6c0f not found: ID does not exist"
Sep 11 12:17:07.891280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.891238    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:17:07.892569 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.892548    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:17:07.970209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970177    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:17:07.970447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970435    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970457    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-metric"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970462    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-metric"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="alertmanager"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970478    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="alertmanager"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970488    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="init-config-reloader"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970493    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="init-config-reloader"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970499    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="config-reloader"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970505    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="config-reloader"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970510    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-web"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970515    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-web"
Sep 11 12:17:07.970517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970523    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="prom-label-proxy"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970528    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="prom-label-proxy"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970568    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-web"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970575    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy-metric"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970582    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="alertmanager"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970588    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="prom-label-proxy"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970594    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="config-reloader"
Sep 11 12:17:07.970848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.970599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" containerName="kube-rbac-proxy"
Sep 11 12:17:07.974206 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.974189    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:07.977462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.977439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 12:17:07.977751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.977728    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 12:17:07.978090 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 12:17:07.978199 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2j67m\""
Sep 11 12:17:07.978199 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 12:17:07.978199 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 12:17:07.978326 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978246    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 12:17:07.978383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978369    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 12:17:07.978769 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.978753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 12:17:07.985255 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:07.985237    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 12:17:08.001313 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.001290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:17:08.083481 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba64768a-0450-4576-ac47-21e2061ac093-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmws\" (UniqueName: \"kubernetes.io/projected/ba64768a-0450-4576-ac47-21e2061ac093-kube-api-access-xrmws\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba64768a-0450-4576-ac47-21e2061ac093-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba64768a-0450-4576-ac47-21e2061ac093-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba64768a-0450-4576-ac47-21e2061ac093-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-web-config\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba64768a-0450-4576-ac47-21e2061ac093-config-out\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.083871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.084227 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.083905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.184900 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.184865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.184900 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.184904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba64768a-0450-4576-ac47-21e2061ac093-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185106 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.184924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmws\" (UniqueName: \"kubernetes.io/projected/ba64768a-0450-4576-ac47-21e2061ac093-kube-api-access-xrmws\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba64768a-0450-4576-ac47-21e2061ac093-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185240 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185294 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185343 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba64768a-0450-4576-ac47-21e2061ac093-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185429 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba64768a-0450-4576-ac47-21e2061ac093-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185494 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185494 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-web-config\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba64768a-0450-4576-ac47-21e2061ac093-config-out\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.185750 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.185715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba64768a-0450-4576-ac47-21e2061ac093-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.186225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.186087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba64768a-0450-4576-ac47-21e2061ac093-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.186712 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.186686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba64768a-0450-4576-ac47-21e2061ac093-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.188301 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.188611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba64768a-0450-4576-ac47-21e2061ac093-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.188837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.188837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.188933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba64768a-0450-4576-ac47-21e2061ac093-config-out\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.188981 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.189019 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.188973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.189019 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.189007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.190469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.190439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba64768a-0450-4576-ac47-21e2061ac093-web-config\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.200488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.200465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmws\" (UniqueName: \"kubernetes.io/projected/ba64768a-0450-4576-ac47-21e2061ac093-kube-api-access-xrmws\") pod \"alertmanager-main-0\" (UID: \"ba64768a-0450-4576-ac47-21e2061ac093\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.284205 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.284171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 12:17:08.575191 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.575167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 12:17:08.576649 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:17:08.576627    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba64768a_0450_4576_ac47_21e2061ac093.slice/crio-a2a6def4fd958cbb63913f7f0ea3ba9f84ad97cf051cf5141f0f59f57ab4ff83 WatchSource:0}: Error finding container a2a6def4fd958cbb63913f7f0ea3ba9f84ad97cf051cf5141f0f59f57ab4ff83: Status 404 returned error can't find the container with id a2a6def4fd958cbb63913f7f0ea3ba9f84ad97cf051cf5141f0f59f57ab4ff83
Sep 11 12:17:08.788457 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.788390    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba64768a-0450-4576-ac47-21e2061ac093" containerID="fef61b859d1730da35521b53442f785d04a57117ec877b4df453a2ab440e976f" exitCode=0
Sep 11 12:17:08.788457 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.788432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerDied","Data":"fef61b859d1730da35521b53442f785d04a57117ec877b4df453a2ab440e976f"}
Sep 11 12:17:08.788763 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:08.788498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"a2a6def4fd958cbb63913f7f0ea3ba9f84ad97cf051cf5141f0f59f57ab4ff83"}
Sep 11 12:17:09.105129 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.105094    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13038ad5-116c-4a19-91a0-ad2d2ece1600" path="/var/lib/kubelet/pods/13038ad5-116c-4a19-91a0-ad2d2ece1600/volumes"
Sep 11 12:17:09.798603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.798570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"e438660bed4f538742e17696a543d873e2dfd6eb2a7a98b64a44f003ae9beeae"}
Sep 11 12:17:09.798762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.798633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"b1ba913b363aa9974788b9ae8c1193eca69e94320f50bd1aefcb6657d46719d4"}
Sep 11 12:17:09.798762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.798659    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"e242375d09b9736b6b6cb2b25bfadb841476e4dd11fa972d4d4acf7e4b1cfda4"}
Sep 11 12:17:09.798762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.798724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"7e27df652efa3a889283e9fc7d8136cb75ffd2dd182b64a81c5a5350d5bd3cc2"}
Sep 11 12:17:09.798762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.798751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"283e25f9a1d06ff97b06a912f72bcd7554ab90559600b1e91859484ae11aaf80"}
Sep 11 12:17:09.798893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.798773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba64768a-0450-4576-ac47-21e2061ac093","Type":"ContainerStarted","Data":"b91421640d06b094b04d0b7098cb5cabce23b603c8ebe9cbb58cdcc53bdd51d9"}
Sep 11 12:17:09.863135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:09.863076    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.849734599 podStartE2EDuration="2.863060255s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.325656ms" isStatefulPod=false podCreationTimestamp="2026-09-11 12:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:17:09.861884763 +0000 UTC m=+165.252422130" watchObservedRunningTime="2026-09-11 12:17:09.863060255 +0000 UTC m=+165.253597596"
Sep 11 12:17:10.219751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.219720    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:17:10.220180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.220009    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="prometheus" containerID="cri-o://102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2" gracePeriod=600
Sep 11 12:17:10.220180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.220029    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy" containerID="cri-o://cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12" gracePeriod=600
Sep 11 12:17:10.220180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.220054    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-thanos" containerID="cri-o://54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf" gracePeriod=600
Sep 11 12:17:10.220180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.220069    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="thanos-sidecar" containerID="cri-o://2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579" gracePeriod=600
Sep 11 12:17:10.220180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.220093    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="config-reloader" containerID="cri-o://e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24" gracePeriod=600
Sep 11 12:17:10.220180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.220069    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-web" containerID="cri-o://bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e" gracePeriod=600
Sep 11 12:17:10.455624 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.455595    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:10.605371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605339    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-kubelet-serving-ca-bundle\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-grpc-tls\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605435    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-web-config\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-metrics-client-ca\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605485    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-kube-rbac-proxy\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605544    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-tls-assets\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605565    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-serving-certs-ca-bundle\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-2\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605636    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-config\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-1\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-metrics-client-certs\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605716    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-config-out\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605761    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605803    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-db\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-0\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605864    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-tls\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.605919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605913    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-kube-api-access-2w9rq\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.606449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-thanos-prometheus-http-client-file\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.606449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-trusted-ca-bundle\") pod \"a9022825-1fa0-4d79-9784-9e5fc624a245\" (UID: \"a9022825-1fa0-4d79-9784-9e5fc624a245\") "
Sep 11 12:17:10.606449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.606207    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.607353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.605938    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.607353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.606566    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.607353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.607243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.607353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.607347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.607723 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.607694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.608264 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.608234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:10.608355 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.608322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.608687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.608645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:17:10.609088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.609061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.609729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.609697    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-config-out" (OuterVolumeSpecName: "config-out") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:17:10.609815 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.609762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-kube-api-access-2w9rq" (OuterVolumeSpecName: "kube-api-access-2w9rq") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "kube-api-access-2w9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:17:10.610041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.609941    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.610041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.610008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-config" (OuterVolumeSpecName: "config") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.610241 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.610217    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.610474 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.610453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.610643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.610612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.610643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.610618    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:17:10.611066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.611045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.617946 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.617926    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-web-config" (OuterVolumeSpecName: "web-config") pod "a9022825-1fa0-4d79-9784-9e5fc624a245" (UID: "a9022825-1fa0-4d79-9784-9e5fc624a245"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:17:10.707245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707218    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-db\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707241    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707252    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707260    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9rq\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-kube-api-access-2w9rq\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707269    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707280    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707288    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-grpc-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707298    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-web-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707305    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-metrics-client-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707314    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-kube-rbac-proxy\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707322    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707331    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9022825-1fa0-4d79-9784-9e5fc624a245-tls-assets\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707339    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707348    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707356    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707365    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9022825-1fa0-4d79-9784-9e5fc624a245-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707373    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-metrics-client-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707382    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9022825-1fa0-4d79-9784-9e5fc624a245-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.707437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.707390    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9022825-1fa0-4d79-9784-9e5fc624a245-config-out\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:17:10.806570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806541    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf" exitCode=0
Sep 11 12:17:10.806570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806562    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12" exitCode=0
Sep 11 12:17:10.806570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806570    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e" exitCode=0
Sep 11 12:17:10.806570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806577    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579" exitCode=0
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806583    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24" exitCode=0
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806588    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2" exitCode=0
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806630    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"}
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"}
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"}
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"}
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806831    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"}
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"}
Sep 11 12:17:10.806918 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.806889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9022825-1fa0-4d79-9784-9e5fc624a245","Type":"ContainerDied","Data":"ee281b50da3d8463d652aaf875bcc87db2995003c6bd19d3522552739769b7aa"}
Sep 11 12:17:10.814327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.814305    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.821171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.821123    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.827627 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.827610    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.833791 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.833770    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.840144 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.840129    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.844190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.844168    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:17:10.847724 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.847354    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.853175 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.850992    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:17:10.863994 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.863973    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.864252 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.864235    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.864297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.864262    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"} err="failed to get container status \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist"
Sep 11 12:17:10.864297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.864280    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.864559 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.864541    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.864608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.864563    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"} err="failed to get container status \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist"
Sep 11 12:17:10.864608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.864580    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.864806 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.864786    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.864846 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.864813    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"} err="failed to get container status \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist"
Sep 11 12:17:10.864846 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.864827    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.865062 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.865047    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.865107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865063    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"} err="failed to get container status \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist"
Sep 11 12:17:10.865107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865074    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.865290 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.865277    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.865326 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865293    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"} err="failed to get container status \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist"
Sep 11 12:17:10.865326 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865305    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.865548 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.865530    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.865615 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865553    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"} err="failed to get container status \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist"
Sep 11 12:17:10.865615 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865565    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.865796 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:17:10.865780    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.865837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865800    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"} err="failed to get container status \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist"
Sep 11 12:17:10.865837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.865813    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.866040 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866025    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"} err="failed to get container status \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist"
Sep 11 12:17:10.866081 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866041    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.866278 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866257    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"} err="failed to get container status \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist"
Sep 11 12:17:10.866355 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866278    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.866534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866517    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"} err="failed to get container status \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist"
Sep 11 12:17:10.866586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866535    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.866753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866737    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"} err="failed to get container status \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist"
Sep 11 12:17:10.866797 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866753    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.866974 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866956    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"} err="failed to get container status \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist"
Sep 11 12:17:10.867040 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.866976    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.867175 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867160    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"} err="failed to get container status \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist"
Sep 11 12:17:10.867220 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867175    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.867388 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867371    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"} err="failed to get container status \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist"
Sep 11 12:17:10.867462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867388    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.867600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867582    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"} err="failed to get container status \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist"
Sep 11 12:17:10.867658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867602    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.867830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867815    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"} err="failed to get container status \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist"
Sep 11 12:17:10.867878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.867831    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.868032 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868012    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"} err="failed to get container status \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist"
Sep 11 12:17:10.868074 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868033    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.868259 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868241    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"} err="failed to get container status \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist"
Sep 11 12:17:10.868318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868260    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.868480 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868462    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"} err="failed to get container status \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist"
Sep 11 12:17:10.868480 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868479    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.868703 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868686    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"} err="failed to get container status \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist"
Sep 11 12:17:10.868747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868703    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.868939 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868922    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"} err="failed to get container status \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist"
Sep 11 12:17:10.868974 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.868941    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.869150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869131    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"} err="failed to get container status \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist"
Sep 11 12:17:10.869226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869150    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.869427 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869409    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"} err="failed to get container status \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist"
Sep 11 12:17:10.869478 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869427    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.869639 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"} err="failed to get container status \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist"
Sep 11 12:17:10.869705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869640    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.869874 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869856    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"} err="failed to get container status \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist"
Sep 11 12:17:10.869917 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.869875    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.870074 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870057    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"} err="failed to get container status \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist"
Sep 11 12:17:10.870140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870076    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.870298 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870279    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"} err="failed to get container status \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist"
Sep 11 12:17:10.870343 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870299    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.870525 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870507    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"} err="failed to get container status \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist"
Sep 11 12:17:10.870596 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870526    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.870735 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870720    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"} err="failed to get container status \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist"
Sep 11 12:17:10.870777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870735    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.870964 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870948    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"} err="failed to get container status \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist"
Sep 11 12:17:10.871029 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.870965    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.871169 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871152    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"} err="failed to get container status \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist"
Sep 11 12:17:10.871219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871170    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.871382 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871365    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"} err="failed to get container status \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist"
Sep 11 12:17:10.871466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871383    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.871611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871595    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"} err="failed to get container status \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist"
Sep 11 12:17:10.871658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871611    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.871835 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871819    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"} err="failed to get container status \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist"
Sep 11 12:17:10.871898 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.871836    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.872067 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872050    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"} err="failed to get container status \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist"
Sep 11 12:17:10.872111 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872067    2561 scope.go:117] "RemoveContainer" containerID="54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"
Sep 11 12:17:10.872271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872251    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf"} err="failed to get container status \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": rpc error: code = NotFound desc = could not find container \"54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf\": container with ID starting with 54a3cdf5a6f294b6b51bd7ec0883f047da397d2c82e0b28d26bbd9c610380ccf not found: ID does not exist"
Sep 11 12:17:10.872340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872272    2561 scope.go:117] "RemoveContainer" containerID="cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"
Sep 11 12:17:10.872518 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872501    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12"} err="failed to get container status \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": rpc error: code = NotFound desc = could not find container \"cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12\": container with ID starting with cfacf441a476363df0b6d3534f96f28fb7acd035b0324ccb0d90532fd61b9a12 not found: ID does not exist"
Sep 11 12:17:10.872581 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872519    2561 scope.go:117] "RemoveContainer" containerID="bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"
Sep 11 12:17:10.872733 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872717    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e"} err="failed to get container status \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": rpc error: code = NotFound desc = could not find container \"bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e\": container with ID starting with bed8f8162c14625da02de62c373aaaed1eee9fbdf68149ea49c9ac5b9495882e not found: ID does not exist"
Sep 11 12:17:10.872778 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872733    2561 scope.go:117] "RemoveContainer" containerID="2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"
Sep 11 12:17:10.872957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872939    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579"} err="failed to get container status \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": rpc error: code = NotFound desc = could not find container \"2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579\": container with ID starting with 2aaa9f8b63cbd417e374d8d6c72c8f72baeeec5d36670c8f4d4ce38bfe20f579 not found: ID does not exist"
Sep 11 12:17:10.873022 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.872958    2561 scope.go:117] "RemoveContainer" containerID="e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"
Sep 11 12:17:10.873173 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.873158    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24"} err="failed to get container status \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": rpc error: code = NotFound desc = could not find container \"e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24\": container with ID starting with e278bbc0226ddbfc24a765e7e24efc85b3048f10674c3812cf2dd666eb424e24 not found: ID does not exist"
Sep 11 12:17:10.873215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.873172    2561 scope.go:117] "RemoveContainer" containerID="102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"
Sep 11 12:17:10.873359 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.873342    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2"} err="failed to get container status \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": rpc error: code = NotFound desc = could not find container \"102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2\": container with ID starting with 102459c1d4abb96775a53304ec62d90858fa8e677814d685e6249fa74d801ed2 not found: ID does not exist"
Sep 11 12:17:10.873433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.873360    2561 scope.go:117] "RemoveContainer" containerID="3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"
Sep 11 12:17:10.873613 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.873596    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8"} err="failed to get container status \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": rpc error: code = NotFound desc = could not find container \"3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8\": container with ID starting with 3e171499fecc5d981508bdc18a7b7f5a4c7f4bc549b05303750ff2a6dbb5d2a8 not found: ID does not exist"
Sep 11 12:17:10.917747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.917727    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:17:10.917994 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.917983    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="init-config-reloader"
Sep 11 12:17:10.918031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.917996    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="init-config-reloader"
Sep 11 12:17:10.918031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918009    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="prometheus"
Sep 11 12:17:10.918031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918014    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="prometheus"
Sep 11 12:17:10.918031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918021    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="thanos-sidecar"
Sep 11 12:17:10.918031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918026    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="thanos-sidecar"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918033    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-thanos"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918038    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-thanos"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918045    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-web"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918051    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-web"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918058    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918063    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918074    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="config-reloader"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918079    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="config-reloader"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918124    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918131    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="prometheus"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918138    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-web"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918144    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="config-reloader"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918151    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="thanos-sidecar"
Sep 11 12:17:10.918188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.918155    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" containerName="kube-rbac-proxy-thanos"
Sep 11 12:17:10.923181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.923166    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:10.929649 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.929627    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 12:17:10.929748 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.929654    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 12:17:10.929838 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.929821    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 12:17:10.930003 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.929987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 12:17:10.930064 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.930048    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 12:17:10.930120 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.930061    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 12:17:10.930792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.930777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bloc51pfupm6d\""
Sep 11 12:17:10.930916 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.930902    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-px227\""
Sep 11 12:17:10.931278 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.931265    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 12:17:10.931350 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.931281    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 12:17:10.931912 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.931897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 12:17:10.932246 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.932231    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 12:17:10.934497 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.934482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 12:17:10.934618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.934603    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 12:17:10.934684 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.934606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 12:17:10.937419 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.937388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 12:17:10.939885 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:10.939869    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 12:17:11.009842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.009814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.009964 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.009848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.009964 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.009873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.009964 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.009946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.009973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmp7\" (UniqueName: \"kubernetes.io/projected/36f90cee-68d3-48ff-b50f-6a8565838916-kube-api-access-wlmp7\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-web-config\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-config\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36f90cee-68d3-48ff-b50f-6a8565838916-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.010479 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.010331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36f90cee-68d3-48ff-b50f-6a8565838916-config-out\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.018922 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.018893    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:17:11.105453 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.105421    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9022825-1fa0-4d79-9784-9e5fc624a245" path="/var/lib/kubelet/pods/a9022825-1fa0-4d79-9784-9e5fc624a245/volumes"
Sep 11 12:17:11.110643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-config\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36f90cee-68d3-48ff-b50f-6a8565838916-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110945 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36f90cee-68d3-48ff-b50f-6a8565838916-config-out\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110945 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110945 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.110945 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.110994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmp7\" (UniqueName: \"kubernetes.io/projected/36f90cee-68d3-48ff-b50f-6a8565838916-kube-api-access-wlmp7\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-web-config\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.111729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.111625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.114378 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.114061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.114566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.114390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.114566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.114393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.114566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.114459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115218 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.114770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115218 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.114880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36f90cee-68d3-48ff-b50f-6a8565838916-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115218 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.115177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-config\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.115266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.115291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.115507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36f90cee-68d3-48ff-b50f-6a8565838916-config-out\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.115558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.115515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.116645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.116627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.116806 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.116788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-web-config\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.116889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.116868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.116973 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.116954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/36f90cee-68d3-48ff-b50f-6a8565838916-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.117239 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.117223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/36f90cee-68d3-48ff-b50f-6a8565838916-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.123474 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.123455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmp7\" (UniqueName: \"kubernetes.io/projected/36f90cee-68d3-48ff-b50f-6a8565838916-kube-api-access-wlmp7\") pod \"prometheus-k8s-0\" (UID: \"36f90cee-68d3-48ff-b50f-6a8565838916\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.231842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.231803    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:11.525104 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.525078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 12:17:11.526542 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:17:11.526514    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f90cee_68d3_48ff_b50f_6a8565838916.slice/crio-799883b009ff54576fada3313c08238d5385de61db705dcc589c565ca53676af WatchSource:0}: Error finding container 799883b009ff54576fada3313c08238d5385de61db705dcc589c565ca53676af: Status 404 returned error can't find the container with id 799883b009ff54576fada3313c08238d5385de61db705dcc589c565ca53676af
Sep 11 12:17:11.812122 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.812045    2561 generic.go:356] "Generic (PLEG): container finished" podID="36f90cee-68d3-48ff-b50f-6a8565838916" containerID="605227c8c97f63b3c25f5d613354272b05ac39dcdf6c41ec22988cba73ca8ef1" exitCode=0
Sep 11 12:17:11.812254 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.812136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerDied","Data":"605227c8c97f63b3c25f5d613354272b05ac39dcdf6c41ec22988cba73ca8ef1"}
Sep 11 12:17:11.812254 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:11.812213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"799883b009ff54576fada3313c08238d5385de61db705dcc589c565ca53676af"}
Sep 11 12:17:12.820389 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.820351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"4e0c588b3726056dbfcff518a9a4221e95f795bd60847b853bcfede40921e099"}
Sep 11 12:17:12.820792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.820449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"dd0a9af2f30937bb20994aff9e8984e12473e1e45a9017928398af259f9c1785"}
Sep 11 12:17:12.820792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.820493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"7baf57e0dc6771b6a4c4fa43e7775e0fab7fe222137762c04adcb681a35eb0fe"}
Sep 11 12:17:12.820792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.820532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"13cf28369aba564de77410add405a94af13f1313d42f72ef595a87076112a12d"}
Sep 11 12:17:12.820792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.820567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"49592c906417cbc204c437976b7f7e9c0663a6b0f14c5a0e9b74eaf888aeaa62"}
Sep 11 12:17:12.820792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.820605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36f90cee-68d3-48ff-b50f-6a8565838916","Type":"ContainerStarted","Data":"f0428308c537c878d57ed3b9406f85e19f00e28d3854132ce5e0952bfbe33576"}
Sep 11 12:17:12.859560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:12.859502    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.840265719 podStartE2EDuration="2.859484342s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.218623ms" isStatefulPod=false podCreationTimestamp="2026-09-11 12:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:17:12.857197928 +0000 UTC m=+168.247735269" watchObservedRunningTime="2026-09-11 12:17:12.859484342 +0000 UTC m=+168.250021685"
Sep 11 12:17:14.827069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:14.827036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:14.885188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:14.885152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:16.172934 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:16.172898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:16.231269 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:16.231237    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:16.232037 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:16.232010    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:17:26.059451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:26.059414    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:26.166236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:26.166201    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:26.240703 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:26.240670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:17:54.561600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:17:54.561559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:03.317487 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:03.317451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:03.376164 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:03.376137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:11.231978 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:11.231947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:18:11.246851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:11.246827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:18:12.066283 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:12.066247    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 12:18:19.945196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:19.945165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:20.048742 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:20.048711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:31.880911 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:31.880876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:31.934661 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:31.934626    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:33.747018 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:33.746990    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:33.801876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:33.801843    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:33.897202 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:33.897166    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:33.952601 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:33.952572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:34.514038 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:34.514001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:34.577580 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:34.577546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:45.110611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:45.110579    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:45.211168 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:45.211136    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:45.310620 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:45.310594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:18:45.364336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:18:45.364263    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:19:03.827664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:03.827623    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:19:03.903692 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:03.903659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:19:25.031279 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:25.031251    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 12:19:32.993063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:32.993019    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:19:34.598669 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:34.598640    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:19:45.043604 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:45.043558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:19:59.523314 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:19:59.523276    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:00.700259 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:00.700221    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:03.326233 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:03.326188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:03.422353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:03.422325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:04.937007 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:04.936971    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:05.034122 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:05.034095    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:05.647186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:05.647154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:05.735226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:05.735188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:08.017863 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:08.017830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:08.253754 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:08.253668    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:10.325148 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:10.325117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:10.427300 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:10.427264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:15.813216 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:15.813178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:17.216556 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:17.216467    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:17.277417 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:17.277362    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:17.812657 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:17.812622    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:17.914320 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:17.914284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:19.627441 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.627391    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"]
Sep 11 12:20:19.630557 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.630533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.633047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.633025    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hqwh4\""
Sep 11 12:20:19.633177 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.633062    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 12:20:19.633750 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.633729    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:20:19.641302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.641278    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"]
Sep 11 12:20:19.740006 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.739978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzk82\" (UniqueName: \"kubernetes.io/projected/e8b3ad9f-2ff2-43ab-976e-73288125a142-kube-api-access-rzk82\") pod \"openshift-lws-operator-fd8ccff4c-vvcm8\" (UID: \"e8b3ad9f-2ff2-43ab-976e-73288125a142\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.740119 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.740017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b3ad9f-2ff2-43ab-976e-73288125a142-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vvcm8\" (UID: \"e8b3ad9f-2ff2-43ab-976e-73288125a142\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.840681 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.840652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk82\" (UniqueName: \"kubernetes.io/projected/e8b3ad9f-2ff2-43ab-976e-73288125a142-kube-api-access-rzk82\") pod \"openshift-lws-operator-fd8ccff4c-vvcm8\" (UID: \"e8b3ad9f-2ff2-43ab-976e-73288125a142\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.840814 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.840690    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b3ad9f-2ff2-43ab-976e-73288125a142-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vvcm8\" (UID: \"e8b3ad9f-2ff2-43ab-976e-73288125a142\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.841100 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.841082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b3ad9f-2ff2-43ab-976e-73288125a142-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vvcm8\" (UID: \"e8b3ad9f-2ff2-43ab-976e-73288125a142\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.849162 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.849136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzk82\" (UniqueName: \"kubernetes.io/projected/e8b3ad9f-2ff2-43ab-976e-73288125a142-kube-api-access-rzk82\") pod \"openshift-lws-operator-fd8ccff4c-vvcm8\" (UID: \"e8b3ad9f-2ff2-43ab-976e-73288125a142\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:19.939359 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:19.939293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"
Sep 11 12:20:20.209070 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:20.209044    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8"]
Sep 11 12:20:20.211533 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:20:20.211506    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8b3ad9f_2ff2_43ab_976e_73288125a142.slice/crio-cd4dd778315fa6268b98a067c5b92186e8497116745e74421f12d5deafcf59e7 WatchSource:0}: Error finding container cd4dd778315fa6268b98a067c5b92186e8497116745e74421f12d5deafcf59e7: Status 404 returned error can't find the container with id cd4dd778315fa6268b98a067c5b92186e8497116745e74421f12d5deafcf59e7
Sep 11 12:20:20.213005 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:20.212988    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:20:20.504656 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:20.504578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8" event={"ID":"e8b3ad9f-2ff2-43ab-976e-73288125a142","Type":"ContainerStarted","Data":"cd4dd778315fa6268b98a067c5b92186e8497116745e74421f12d5deafcf59e7"}
Sep 11 12:20:23.520286 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:23.520249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8" event={"ID":"e8b3ad9f-2ff2-43ab-976e-73288125a142","Type":"ContainerStarted","Data":"ed7dd6f46f0f07da687af4d80f125e3ac9796e78b4559fe437e7d20491623976"}
Sep 11 12:20:23.537089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:23.537038    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vvcm8" podStartSLOduration=2.077599541 podStartE2EDuration="4.537021974s" totalImagesPullingTime="2.459422433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:23.535204875 +0000 UTC m=+358.925742216" watchObservedRunningTime="2026-09-11 12:20:23.537021974 +0000 UTC m=+358.927559315"
Sep 11 12:20:24.221695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:24.221658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:24.323291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:24.323252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:26.230310 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:26.230274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:38.947864 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:38.947829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:39.041392 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:39.041351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:39.557393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:39.557355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:39.848221 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:39.848186    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:39.958798 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:39.958768    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.003919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.003880    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"]
Sep 11 12:20:40.007427 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.007389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.010463 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.010442    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n5jwr\""
Sep 11 12:20:40.010708 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.010692    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 12:20:40.010775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.010756    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 12:20:40.011309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.011294    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 12:20:40.012682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.012662    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 12:20:40.034196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.034167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"]
Sep 11 12:20:40.104579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.104524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3686c6d-9ad6-4114-ac21-c629251fd600-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.104579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.104564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3686c6d-9ad6-4114-ac21-c629251fd600-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.104714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.104626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3686c6d-9ad6-4114-ac21-c629251fd600-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.104714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.104654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3686c6d-9ad6-4114-ac21-c629251fd600-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.104714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.104679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbmj\" (UniqueName: \"kubernetes.io/projected/e3686c6d-9ad6-4114-ac21-c629251fd600-kube-api-access-ctbmj\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.177755 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.177725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.205719 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.205694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3686c6d-9ad6-4114-ac21-c629251fd600-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.205808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.205743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3686c6d-9ad6-4114-ac21-c629251fd600-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.205808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.205767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3686c6d-9ad6-4114-ac21-c629251fd600-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.205808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.205789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbmj\" (UniqueName: \"kubernetes.io/projected/e3686c6d-9ad6-4114-ac21-c629251fd600-kube-api-access-ctbmj\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.205921 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.205813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3686c6d-9ad6-4114-ac21-c629251fd600-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.206202 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.206177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3686c6d-9ad6-4114-ac21-c629251fd600-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.208186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.208156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3686c6d-9ad6-4114-ac21-c629251fd600-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.208278 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.208229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3686c6d-9ad6-4114-ac21-c629251fd600-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.208319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.208294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3686c6d-9ad6-4114-ac21-c629251fd600-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.214478 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.214458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbmj\" (UniqueName: \"kubernetes.io/projected/e3686c6d-9ad6-4114-ac21-c629251fd600-kube-api-access-ctbmj\") pod \"opendatahub-operator-controller-manager-8dd64745b-k56gw\" (UID: \"e3686c6d-9ad6-4114-ac21-c629251fd600\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.291889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.291853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.316783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.316763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:40.381373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.381332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.480032 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.480000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.570094 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.570062    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.593536 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.593508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"]
Sep 11 12:20:40.596517 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:20:40.596494    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3686c6d_9ad6_4114_ac21_c629251fd600.slice/crio-d40f77651c65ceac06469978d65f6c09061d359e844474baf388a81aef1dcbb8 WatchSource:0}: Error finding container d40f77651c65ceac06469978d65f6c09061d359e844474baf388a81aef1dcbb8: Status 404 returned error can't find the container with id d40f77651c65ceac06469978d65f6c09061d359e844474baf388a81aef1dcbb8
Sep 11 12:20:40.627312 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.627248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.728097 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.728063    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:40.831842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:40.831810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:41.588222 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:41.588172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw" event={"ID":"e3686c6d-9ad6-4114-ac21-c629251fd600","Type":"ContainerStarted","Data":"d40f77651c65ceac06469978d65f6c09061d359e844474baf388a81aef1dcbb8"}
Sep 11 12:20:44.601817 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:44.601776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw" event={"ID":"e3686c6d-9ad6-4114-ac21-c629251fd600","Type":"ContainerStarted","Data":"f6b0f5ca75fd8519fbb1e704635c21e110e57374e7cab5df0f0955b7777de9d9"}
Sep 11 12:20:45.607568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:45.607532    2561 generic.go:356] "Generic (PLEG): container finished" podID="e3686c6d-9ad6-4114-ac21-c629251fd600" containerID="f6b0f5ca75fd8519fbb1e704635c21e110e57374e7cab5df0f0955b7777de9d9" exitCode=0
Sep 11 12:20:45.607568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:45.607574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw" event={"ID":"e3686c6d-9ad6-4114-ac21-c629251fd600","Type":"ContainerDied","Data":"f6b0f5ca75fd8519fbb1e704635c21e110e57374e7cab5df0f0955b7777de9d9"}
Sep 11 12:20:46.613014 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:46.612974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw" event={"ID":"e3686c6d-9ad6-4114-ac21-c629251fd600","Type":"ContainerStarted","Data":"1dd48fbaaffbbc7494bc025178f8af2acb93f459ad491c34a61f88dba5213b1d"}
Sep 11 12:20:46.613381 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:46.613109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:46.636065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:46.636015    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw" podStartSLOduration=3.286651065 podStartE2EDuration="7.636002389s" totalImagesPullingTime="3.35184268s" totalInitContainerRuntime="997.508644ms" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:46.634969624 +0000 UTC m=+382.025506957" watchObservedRunningTime="2026-09-11 12:20:46.636002389 +0000 UTC m=+382.026539731"
Sep 11 12:20:51.284973 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:51.284933    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:51.387356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:51.387318    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:57.619297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:57.619267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-k56gw"
Sep 11 12:20:59.405538 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:59.405506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:59.515609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:59.515576    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:20:59.966329 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:20:59.966290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:00.074747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:00.074717    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:00.618689 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:00.618655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:01.544004 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:01.543964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:01.634251 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:01.634219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:02.918171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:02.918134    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:04.772501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:04.772466    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:06.319187 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:06.319152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:06.547567 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:06.547532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:06.723463 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:06.723429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:06.846392 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:06.846358    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:06.940469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:06.940391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:08.287102 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:08.287014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:08.349690 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:08.349645    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.352086 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.352051    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.419413 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.419348    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.565952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.565918    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.694757 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.694675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.797546 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.797508    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.859297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.859256    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:10.914455 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:10.914423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:11.036866 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:11.036780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:18.661272 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.661236    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"]
Sep 11 12:21:18.663914 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.663890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.666881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.666856    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 12:21:18.668791 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.668776    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7gndt\""
Sep 11 12:21:18.671116 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.671097    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 12:21:18.671192 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.671150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 12:21:18.686289 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.686267    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"]
Sep 11 12:21:18.721224 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.721197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74fa9b2d-13d8-4277-8509-5c25e3b2772b-manager-config\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.721331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.721234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74fa9b2d-13d8-4277-8509-5c25e3b2772b-cert\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.721331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.721288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74fa9b2d-13d8-4277-8509-5c25e3b2772b-metrics-cert\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.721331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.721306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvtrr\" (UniqueName: \"kubernetes.io/projected/74fa9b2d-13d8-4277-8509-5c25e3b2772b-kube-api-access-dvtrr\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.822742 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.822703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74fa9b2d-13d8-4277-8509-5c25e3b2772b-manager-config\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.822941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.822764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74fa9b2d-13d8-4277-8509-5c25e3b2772b-cert\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.822941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.822814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74fa9b2d-13d8-4277-8509-5c25e3b2772b-metrics-cert\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.822941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.822841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvtrr\" (UniqueName: \"kubernetes.io/projected/74fa9b2d-13d8-4277-8509-5c25e3b2772b-kube-api-access-dvtrr\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.823542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.823509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74fa9b2d-13d8-4277-8509-5c25e3b2772b-manager-config\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.825970 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.825936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74fa9b2d-13d8-4277-8509-5c25e3b2772b-cert\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.826367 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.826345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74fa9b2d-13d8-4277-8509-5c25e3b2772b-metrics-cert\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.833611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.833585    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvtrr\" (UniqueName: \"kubernetes.io/projected/74fa9b2d-13d8-4277-8509-5c25e3b2772b-kube-api-access-dvtrr\") pod \"lws-controller-manager-69c967fbf8-mzdtf\" (UID: \"74fa9b2d-13d8-4277-8509-5c25e3b2772b\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:18.972984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:18.972903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:19.259113 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:19.259038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"]
Sep 11 12:21:19.263051 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:21:19.263018    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fa9b2d_13d8_4277_8509_5c25e3b2772b.slice/crio-495def1f985885528488acfcc58eaa00110a7d96b3c01c91012290ef1d6876f9 WatchSource:0}: Error finding container 495def1f985885528488acfcc58eaa00110a7d96b3c01c91012290ef1d6876f9: Status 404 returned error can't find the container with id 495def1f985885528488acfcc58eaa00110a7d96b3c01c91012290ef1d6876f9
Sep 11 12:21:19.744666 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:19.744636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf" event={"ID":"74fa9b2d-13d8-4277-8509-5c25e3b2772b","Type":"ContainerStarted","Data":"495def1f985885528488acfcc58eaa00110a7d96b3c01c91012290ef1d6876f9"}
Sep 11 12:21:21.252275 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.252196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:21.365816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.365784    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:21.417912 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.417881    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"]
Sep 11 12:21:21.426497 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.421387    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"]
Sep 11 12:21:21.426497 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.422819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:21.426885 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.426867    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.427879 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.427855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 12:21:21.428013 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.427928    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f4tgz\""
Sep 11 12:21:21.431819 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.431787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 12:21:21.433723 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.433705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m4bcj\""
Sep 11 12:21:21.439120 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.438346    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"]
Sep 11 12:21:21.443452 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.443423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"]
Sep 11 12:21:21.446902 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.446883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7952582-3f47-43c5-99c9-c9f711169df2-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:21.446998 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.446958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9lj\" (UniqueName: \"kubernetes.io/projected/a7952582-3f47-43c5-99c9-c9f711169df2-kube-api-access-ss9lj\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:21.462304 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.462270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:21.547509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.547435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9lj\" (UniqueName: \"kubernetes.io/projected/a7952582-3f47-43c5-99c9-c9f711169df2-kube-api-access-ss9lj\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:21.547509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.547475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7952582-3f47-43c5-99c9-c9f711169df2-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:21.547509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.547504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba0dae-21aa-4501-9a7c-1ff4542a2609-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nwltg\" (UID: \"7eba0dae-21aa-4501-9a7c-1ff4542a2609\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.547700 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.547536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpjq\" (UniqueName: \"kubernetes.io/projected/7eba0dae-21aa-4501-9a7c-1ff4542a2609-kube-api-access-2xpjq\") pod \"llmisvc-controller-manager-777c6bd7d5-nwltg\" (UID: \"7eba0dae-21aa-4501-9a7c-1ff4542a2609\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.547700 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:21:21.547597    2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 12:21:21.547700 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:21:21.547672    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7952582-3f47-43c5-99c9-c9f711169df2-cert podName:a7952582-3f47-43c5-99c9-c9f711169df2 nodeName:}" failed. No retries permitted until 2026-09-11 12:21:22.047655834 +0000 UTC m=+417.438193154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a7952582-3f47-43c5-99c9-c9f711169df2-cert") pod "kserve-controller-manager-d6f4b65b8-jhw8r" (UID: "a7952582-3f47-43c5-99c9-c9f711169df2") : secret "kserve-webhook-server-cert" not found
Sep 11 12:21:21.572894 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.572860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9lj\" (UniqueName: \"kubernetes.io/projected/a7952582-3f47-43c5-99c9-c9f711169df2-kube-api-access-ss9lj\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:21.588833 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.588794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:21.648768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.648739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba0dae-21aa-4501-9a7c-1ff4542a2609-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nwltg\" (UID: \"7eba0dae-21aa-4501-9a7c-1ff4542a2609\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.648869 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.648796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpjq\" (UniqueName: \"kubernetes.io/projected/7eba0dae-21aa-4501-9a7c-1ff4542a2609-kube-api-access-2xpjq\") pod \"llmisvc-controller-manager-777c6bd7d5-nwltg\" (UID: \"7eba0dae-21aa-4501-9a7c-1ff4542a2609\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.651025 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.651005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7eba0dae-21aa-4501-9a7c-1ff4542a2609-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nwltg\" (UID: \"7eba0dae-21aa-4501-9a7c-1ff4542a2609\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.668574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.668550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpjq\" (UniqueName: \"kubernetes.io/projected/7eba0dae-21aa-4501-9a7c-1ff4542a2609-kube-api-access-2xpjq\") pod \"llmisvc-controller-manager-777c6bd7d5-nwltg\" (UID: \"7eba0dae-21aa-4501-9a7c-1ff4542a2609\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.744663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.744637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:21.751107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.751078    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:21.758693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.758670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf" event={"ID":"74fa9b2d-13d8-4277-8509-5c25e3b2772b","Type":"ContainerStarted","Data":"b8c3ae2a5caa1521c107753506df3e896238db9cba9959f69d2911f5e5b0d013"}
Sep 11 12:21:21.758808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.758792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:21.797889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:21.797785    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf" podStartSLOduration=2.23326272 podStartE2EDuration="3.797759109s" totalImagesPullingTime="1.564496389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:21.795212721 +0000 UTC m=+417.185750063" watchObservedRunningTime="2026-09-11 12:21:21.797759109 +0000 UTC m=+417.188296451"
Sep 11 12:21:22.036018 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.035985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"]
Sep 11 12:21:22.039657 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:21:22.039629    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7eba0dae_21aa_4501_9a7c_1ff4542a2609.slice/crio-fb77810a699e6072f5ab008dba49a3597e7593b293e6595f224b0b812a0d8a83 WatchSource:0}: Error finding container fb77810a699e6072f5ab008dba49a3597e7593b293e6595f224b0b812a0d8a83: Status 404 returned error can't find the container with id fb77810a699e6072f5ab008dba49a3597e7593b293e6595f224b0b812a0d8a83
Sep 11 12:21:22.051851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.051795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7952582-3f47-43c5-99c9-c9f711169df2-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:22.054108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.054087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7952582-3f47-43c5-99c9-c9f711169df2-cert\") pod \"kserve-controller-manager-d6f4b65b8-jhw8r\" (UID: \"a7952582-3f47-43c5-99c9-c9f711169df2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:22.335393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.335362    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:22.622980 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.622902    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"]
Sep 11 12:21:22.625758 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:21:22.625728    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7952582_3f47_43c5_99c9_c9f711169df2.slice/crio-c66cfd816c316a248f4b3677dcfde7a83c4958d1b1582410b1ec44a81857329c WatchSource:0}: Error finding container c66cfd816c316a248f4b3677dcfde7a83c4958d1b1582410b1ec44a81857329c: Status 404 returned error can't find the container with id c66cfd816c316a248f4b3677dcfde7a83c4958d1b1582410b1ec44a81857329c
Sep 11 12:21:22.664844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.664818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:22.763461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.763421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r" event={"ID":"a7952582-3f47-43c5-99c9-c9f711169df2","Type":"ContainerStarted","Data":"c66cfd816c316a248f4b3677dcfde7a83c4958d1b1582410b1ec44a81857329c"}
Sep 11 12:21:22.764574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.764552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg" event={"ID":"7eba0dae-21aa-4501-9a7c-1ff4542a2609","Type":"ContainerStarted","Data":"fb77810a699e6072f5ab008dba49a3597e7593b293e6595f224b0b812a0d8a83"}
Sep 11 12:21:22.781143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.781111    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:22.882628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:22.882540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:26.786509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:26.786465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg" event={"ID":"7eba0dae-21aa-4501-9a7c-1ff4542a2609","Type":"ContainerStarted","Data":"706ca6943d5469931fc0d111403f72a19e26ea807f6097bca81529cd309649cf"}
Sep 11 12:21:26.786981 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:26.786600    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:26.787985 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:26.787960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r" event={"ID":"a7952582-3f47-43c5-99c9-c9f711169df2","Type":"ContainerStarted","Data":"dbac02fca8fa37477b49c868dcfb0152db8b695645f556864c54d812e0ac87c7"}
Sep 11 12:21:26.788107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:26.788095    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:21:26.807877 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:26.807834    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg" podStartSLOduration=1.7059761629999999 podStartE2EDuration="5.807821993s" totalImagesPullingTime="4.10184583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:26.807250781 +0000 UTC m=+422.197788124" watchObservedRunningTime="2026-09-11 12:21:26.807821993 +0000 UTC m=+422.198359335"
Sep 11 12:21:26.827783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:26.827743    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r" podStartSLOduration=2.312180362 podStartE2EDuration="5.827730177s" totalImagesPullingTime="3.515549815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:26.826624412 +0000 UTC m=+422.217161755" watchObservedRunningTime="2026-09-11 12:21:26.827730177 +0000 UTC m=+422.218267519"
Sep 11 12:21:32.767065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:32.767032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-mzdtf"
Sep 11 12:21:51.929842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:51.929802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:52.040501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:52.040463    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:53.884369 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:53.884327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:53.936144 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:53.936102    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:21:57.794467 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:57.794429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nwltg"
Sep 11 12:21:57.797919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:21:57.797893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jhw8r"
Sep 11 12:22:11.500448 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:11.500388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:11.602178 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:11.602130    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:30.923830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:30.923789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:31.576332 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:31.576291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:32.724531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:32.724482    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:34.641891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:34.641854    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:34.742386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:34.742354    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:35.990553 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:35.990518    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:37.045044 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.045007    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:37.333150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.333102    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"]
Sep 11 12:22:37.335589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.335573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:37.337937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.337916    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 12:22:37.338043 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.337940    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 12:22:37.338043 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.337959    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 12:22:37.338043 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.337988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jbz5l\""
Sep 11 12:22:37.351957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.351939    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"]
Sep 11 12:22:37.467835 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.467793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2d4\" (UniqueName: \"kubernetes.io/projected/6a4583d8-7f62-401f-aba6-f36cf834061d-kube-api-access-9g2d4\") pod \"dns-operator-controller-manager-7b84cd7c9c-jxx8m\" (UID: \"6a4583d8-7f62-401f-aba6-f36cf834061d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:37.568381 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.568325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2d4\" (UniqueName: \"kubernetes.io/projected/6a4583d8-7f62-401f-aba6-f36cf834061d-kube-api-access-9g2d4\") pod \"dns-operator-controller-manager-7b84cd7c9c-jxx8m\" (UID: \"6a4583d8-7f62-401f-aba6-f36cf834061d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:37.577284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.577262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2d4\" (UniqueName: \"kubernetes.io/projected/6a4583d8-7f62-401f-aba6-f36cf834061d-kube-api-access-9g2d4\") pod \"dns-operator-controller-manager-7b84cd7c9c-jxx8m\" (UID: \"6a4583d8-7f62-401f-aba6-f36cf834061d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:37.645631 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.645541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:37.915409 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:37.915371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"]
Sep 11 12:22:37.917890 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:22:37.917853    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4583d8_7f62_401f_aba6_f36cf834061d.slice/crio-ed25c7d890910b651b4df4a22310fdd25ae971718c8832cb9457fc5ed7b313b8 WatchSource:0}: Error finding container ed25c7d890910b651b4df4a22310fdd25ae971718c8832cb9457fc5ed7b313b8: Status 404 returned error can't find the container with id ed25c7d890910b651b4df4a22310fdd25ae971718c8832cb9457fc5ed7b313b8
Sep 11 12:22:38.091851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:38.091823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m" event={"ID":"6a4583d8-7f62-401f-aba6-f36cf834061d","Type":"ContainerStarted","Data":"ed25c7d890910b651b4df4a22310fdd25ae971718c8832cb9457fc5ed7b313b8"}
Sep 11 12:22:38.188912 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:38.188838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:38.418667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:38.418632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:39.838925 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:39.838889    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"]
Sep 11 12:22:39.845660 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:39.845634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:39.848093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:39.848062    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x5gg9\""
Sep 11 12:22:39.855018 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:39.854890    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"]
Sep 11 12:22:39.989069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:39.989026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwlm\" (UniqueName: \"kubernetes.io/projected/a34483e3-1405-4967-9042-433e800414b4-kube-api-access-lmwlm\") pod \"limitador-operator-controller-manager-74495c69f8-7d628\" (UID: \"a34483e3-1405-4967-9042-433e800414b4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:40.030387 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.030329    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:40.089800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.089713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwlm\" (UniqueName: \"kubernetes.io/projected/a34483e3-1405-4967-9042-433e800414b4-kube-api-access-lmwlm\") pod \"limitador-operator-controller-manager-74495c69f8-7d628\" (UID: \"a34483e3-1405-4967-9042-433e800414b4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:40.098743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.098717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwlm\" (UniqueName: \"kubernetes.io/projected/a34483e3-1405-4967-9042-433e800414b4-kube-api-access-lmwlm\") pod \"limitador-operator-controller-manager-74495c69f8-7d628\" (UID: \"a34483e3-1405-4967-9042-433e800414b4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:40.160187 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.160152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:40.392287 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.392258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:40.594693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.593285    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"]
Sep 11 12:22:40.598681 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:22:40.598648    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34483e3_1405_4967_9042_433e800414b4.slice/crio-556b2567ef9661dedff59df618a77d41dfce251524edcc49b58f2277364f58b0 WatchSource:0}: Error finding container 556b2567ef9661dedff59df618a77d41dfce251524edcc49b58f2277364f58b0: Status 404 returned error can't find the container with id 556b2567ef9661dedff59df618a77d41dfce251524edcc49b58f2277364f58b0
Sep 11 12:22:40.621991 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.621911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:40.746853 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:40.746819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:41.107934 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:41.107899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m" event={"ID":"6a4583d8-7f62-401f-aba6-f36cf834061d","Type":"ContainerStarted","Data":"a2039ecbae63c2933f36cb1a75490dc33f9559ab266a504911d6f2f44b9c4d76"}
Sep 11 12:22:41.108376 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:41.108001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:41.109253 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:41.109228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628" event={"ID":"a34483e3-1405-4967-9042-433e800414b4","Type":"ContainerStarted","Data":"556b2567ef9661dedff59df618a77d41dfce251524edcc49b58f2277364f58b0"}
Sep 11 12:22:41.130169 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:41.130124    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m" podStartSLOduration=1.667377388 podStartE2EDuration="4.130109793s" totalImagesPullingTime="2.462732405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:41.128807137 +0000 UTC m=+496.519344517" watchObservedRunningTime="2026-09-11 12:22:41.130109793 +0000 UTC m=+496.520647135"
Sep 11 12:22:43.120423 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:43.120373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628" event={"ID":"a34483e3-1405-4967-9042-433e800414b4","Type":"ContainerStarted","Data":"b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2"}
Sep 11 12:22:43.120837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:43.120447    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:43.142473 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:43.142418    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628" podStartSLOduration=2.526828827 podStartE2EDuration="4.142384774s" totalImagesPullingTime="1.615555947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:43.141492063 +0000 UTC m=+498.532029406" watchObservedRunningTime="2026-09-11 12:22:43.142384774 +0000 UTC m=+498.532922115"
Sep 11 12:22:43.221730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:43.221699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:43.441490 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:43.441388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:45.594654 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:45.594605    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:45.651151 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:45.651120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:45.988795 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:45.988721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:46.067763 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:46.067726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:46.397625 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:46.397584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:46.459302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:46.459271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:49.744531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:49.744500    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:49.800563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:49.800531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:52.101785 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:52.101753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:52.117564 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:52.117538    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jxx8m"
Sep 11 12:22:52.203917 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:52.203883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:54.126988 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.126959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:54.163579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.163547    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"]
Sep 11 12:22:54.166933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.166910    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.169291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.169265    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tt57v\""
Sep 11 12:22:54.179197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.179172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"]
Sep 11 12:22:54.316749 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.316704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg682\" (UniqueName: \"kubernetes.io/projected/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-kube-api-access-hg682\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.316933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.316853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.417806 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.417722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.417806 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.417768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg682\" (UniqueName: \"kubernetes.io/projected/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-kube-api-access-hg682\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.418132 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.418112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.426124 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.426103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg682\" (UniqueName: \"kubernetes.io/projected/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-kube-api-access-hg682\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.477745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.477720    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.613466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.613434    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"]
Sep 11 12:22:54.626372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.626344    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"]
Sep 11 12:22:54.643741 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.643711    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"]
Sep 11 12:22:54.643962 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.643935    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628" podUID="a34483e3-1405-4967-9042-433e800414b4" containerName="manager" containerID="cri-o://b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2" gracePeriod=2
Sep 11 12:22:54.649740 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.649717    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"]
Sep 11 12:22:54.653482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.653463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.655272 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.655249    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"]
Sep 11 12:22:54.670259 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:54.670191    2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a34483e3-1405-4967-9042-433e800414b4\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-7d628\": pods \"limitador-operator-controller-manager-74495c69f8-7d628\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:54.672787 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:54.672726    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7d628\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="a34483e3-1405-4967-9042-433e800414b4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:54.675438 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.675378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"]
Sep 11 12:22:54.677608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.677572    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"]
Sep 11 12:22:54.678084 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.678045    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34483e3-1405-4967-9042-433e800414b4" containerName="manager"
Sep 11 12:22:54.678084 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.678066    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34483e3-1405-4967-9042-433e800414b4" containerName="manager"
Sep 11 12:22:54.678239 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.678179    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34483e3-1405-4967-9042-433e800414b4" containerName="manager"
Sep 11 12:22:54.681305 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.681286    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:22:54.687988 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.687968    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"]
Sep 11 12:22:54.700942 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:54.700911    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7d628\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="a34483e3-1405-4967-9042-433e800414b4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:54.821160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.820949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twg4p\" (UniqueName: \"kubernetes.io/projected/c894c769-7958-44a3-b384-a2400135da45-kube-api-access-twg4p\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-fvkqq\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.821160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.821009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ms4k\" (UniqueName: \"kubernetes.io/projected/a632f042-67c1-48ee-9e1c-06bb2a196cd7-kube-api-access-5ms4k\") pod \"limitador-operator-controller-manager-74495c69f8-vq5jn\" (UID: \"a632f042-67c1-48ee-9e1c-06bb2a196cd7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:22:54.821160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.821096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c894c769-7958-44a3-b384-a2400135da45-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-fvkqq\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.858893 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:54.858859    2561 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 12:22:54.858893 ip-10-0-135-5 kubenswrapper[2561]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-4vhr7_kuadrant-system_6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9_0(f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-4vhr7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c" Netns:"/var/run/netns/c6a1cb79-ea57-4493-8957-e5938348ac49" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-4vhr7;K8S_POD_INFRA_CONTAINER_ID=f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c;K8S_POD_UID=6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7 f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:19 [10.133.0.25/23]
Sep 11 12:22:54.858893 ip-10-0-135-5 kubenswrapper[2561]:         '
Sep 11 12:22:54.858893 ip-10-0-135-5 kubenswrapper[2561]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 12:22:54.858893 ip-10-0-135-5 kubenswrapper[2561]:  >
Sep 11 12:22:54.859118 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:54.858920    2561 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 12:22:54.859118 ip-10-0-135-5 kubenswrapper[2561]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-4vhr7_kuadrant-system_6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9_0(f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-4vhr7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c" Netns:"/var/run/netns/c6a1cb79-ea57-4493-8957-e5938348ac49" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-4vhr7;K8S_POD_INFRA_CONTAINER_ID=f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c;K8S_POD_UID=6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7 f9d7fdcc550f0693cbcd9a8dbb4778a1b94327db4fae7d3bd104d4bc7310fd0c network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:19 [10.133.0.25/23]
Sep 11 12:22:54.859118 ip-10-0-135-5 kubenswrapper[2561]:         '
Sep 11 12:22:54.859118 ip-10-0-135-5 kubenswrapper[2561]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 12:22:54.859118 ip-10-0-135-5 kubenswrapper[2561]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:54.872315 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.872293    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:54.875164 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:54.875138    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7d628\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="a34483e3-1405-4967-9042-433e800414b4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:54.922621 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.922564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twg4p\" (UniqueName: \"kubernetes.io/projected/c894c769-7958-44a3-b384-a2400135da45-kube-api-access-twg4p\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-fvkqq\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.922621 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.922601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ms4k\" (UniqueName: \"kubernetes.io/projected/a632f042-67c1-48ee-9e1c-06bb2a196cd7-kube-api-access-5ms4k\") pod \"limitador-operator-controller-manager-74495c69f8-vq5jn\" (UID: \"a632f042-67c1-48ee-9e1c-06bb2a196cd7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:22:54.922733 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.922643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c894c769-7958-44a3-b384-a2400135da45-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-fvkqq\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.923009 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.922992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c894c769-7958-44a3-b384-a2400135da45-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-fvkqq\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.936197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.936178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twg4p\" (UniqueName: \"kubernetes.io/projected/c894c769-7958-44a3-b384-a2400135da45-kube-api-access-twg4p\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-fvkqq\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:54.940495 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:54.940480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ms4k\" (UniqueName: \"kubernetes.io/projected/a632f042-67c1-48ee-9e1c-06bb2a196cd7-kube-api-access-5ms4k\") pod \"limitador-operator-controller-manager-74495c69f8-vq5jn\" (UID: \"a632f042-67c1-48ee-9e1c-06bb2a196cd7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:22:55.023517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.023490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmwlm\" (UniqueName: \"kubernetes.io/projected/a34483e3-1405-4967-9042-433e800414b4-kube-api-access-lmwlm\") pod \"a34483e3-1405-4967-9042-433e800414b4\" (UID: \"a34483e3-1405-4967-9042-433e800414b4\") "
Sep 11 12:22:55.025470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.025449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34483e3-1405-4967-9042-433e800414b4-kube-api-access-lmwlm" (OuterVolumeSpecName: "kube-api-access-lmwlm") pod "a34483e3-1405-4967-9042-433e800414b4" (UID: "a34483e3-1405-4967-9042-433e800414b4"). InnerVolumeSpecName "kube-api-access-lmwlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:22:55.044700 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.044679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:55.052482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.052457    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:22:55.108274 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:55.108124    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7d628\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="a34483e3-1405-4967-9042-433e800414b4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:55.110956 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.110930    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34483e3-1405-4967-9042-433e800414b4" path="/var/lib/kubelet/pods/a34483e3-1405-4967-9042-433e800414b4/volumes"
Sep 11 12:22:55.124834 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.124802    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwlm\" (UniqueName: \"kubernetes.io/projected/a34483e3-1405-4967-9042-433e800414b4-kube-api-access-lmwlm\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:22:55.173859 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.173796    2561 generic.go:356] "Generic (PLEG): container finished" podID="a34483e3-1405-4967-9042-433e800414b4" containerID="b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2" exitCode=0
Sep 11 12:22:55.173859 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.173842    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7d628"
Sep 11 12:22:55.174267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.173894    2561 scope.go:117] "RemoveContainer" containerID="b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2"
Sep 11 12:22:55.174267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.174005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:55.182549 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.182535    2561 scope.go:117] "RemoveContainer" containerID="b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2"
Sep 11 12:22:55.182788 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.182769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:55.182878 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:55.182808    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2\": container with ID starting with b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2 not found: ID does not exist" containerID="b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2"
Sep 11 12:22:55.182878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.182832    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2"} err="failed to get container status \"b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2\": rpc error: code = NotFound desc = could not find container \"b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2\": container with ID starting with b2d51749e58452f066628d39f570900793523d5f42e8b643e985a062fa8ff7e2 not found: ID does not exist"
Sep 11 12:22:55.185298 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:55.185274    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:55.326477 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.326448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg682\" (UniqueName: \"kubernetes.io/projected/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-kube-api-access-hg682\") pod \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") "
Sep 11 12:22:55.326616 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.326520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-extensions-socket-volume\") pod \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\" (UID: \"6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9\") "
Sep 11 12:22:55.326803 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.326777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" (UID: "6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:22:55.329004 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.328967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-kube-api-access-hg682" (OuterVolumeSpecName: "kube-api-access-hg682") pod "6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" (UID: "6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9"). InnerVolumeSpecName "kube-api-access-hg682". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:22:55.329854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.329827    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"]
Sep 11 12:22:55.331302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.331283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"]
Sep 11 12:22:55.332672 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:22:55.332645    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc894c769_7958_44a3_b384_a2400135da45.slice/crio-0f06497cdbcf18a9d03e0091f211330a3492c479cc5112087995d882a5cf9004 WatchSource:0}: Error finding container 0f06497cdbcf18a9d03e0091f211330a3492c479cc5112087995d882a5cf9004: Status 404 returned error can't find the container with id 0f06497cdbcf18a9d03e0091f211330a3492c479cc5112087995d882a5cf9004
Sep 11 12:22:55.333557 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:22:55.333536    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda632f042_67c1_48ee_9e1c_06bb2a196cd7.slice/crio-97b55d078743ee25a713e05308106f74226cc1e6b0a8ce8a116e84e4179875a8 WatchSource:0}: Error finding container 97b55d078743ee25a713e05308106f74226cc1e6b0a8ce8a116e84e4179875a8: Status 404 returned error can't find the container with id 97b55d078743ee25a713e05308106f74226cc1e6b0a8ce8a116e84e4179875a8
Sep 11 12:22:55.428190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.428134    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-extensions-socket-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:22:55.428190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:55.428158    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg682\" (UniqueName: \"kubernetes.io/projected/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9-kube-api-access-hg682\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:22:56.180417 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.180370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq" event={"ID":"c894c769-7958-44a3-b384-a2400135da45","Type":"ContainerStarted","Data":"0f06497cdbcf18a9d03e0091f211330a3492c479cc5112087995d882a5cf9004"}
Sep 11 12:22:56.182553 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.182517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn" event={"ID":"a632f042-67c1-48ee-9e1c-06bb2a196cd7","Type":"ContainerStarted","Data":"8fdf1f44bf50e04f11441841d095174105fefa44dd7a9705d0159ffa35ed8a0a"}
Sep 11 12:22:56.182697 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.182570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn" event={"ID":"a632f042-67c1-48ee-9e1c-06bb2a196cd7","Type":"ContainerStarted","Data":"97b55d078743ee25a713e05308106f74226cc1e6b0a8ce8a116e84e4179875a8"}
Sep 11 12:22:56.182697 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.182609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:22:56.183797 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.183776    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:56.184748 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:56.184719    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:56.206631 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:22:56.206593    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-4vhr7\" is forbidden: User \"system:node:ip-10-0-135-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-5.ec2.internal' and this object" podUID="6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-4vhr7"
Sep 11 12:22:56.842749 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.842711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:56.953201 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:56.953148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:22:57.107276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:57.107197    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9" path="/var/lib/kubelet/pods/6be3b946-7cb1-4ba8-80a4-aa62eb4fa7f9/volumes"
Sep 11 12:22:59.201297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:59.201265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq" event={"ID":"c894c769-7958-44a3-b384-a2400135da45","Type":"ContainerStarted","Data":"a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459"}
Sep 11 12:22:59.201672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:59.201346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:22:59.224329 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:59.224285    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq" podStartSLOduration=1.8591684179999999 podStartE2EDuration="5.224270076s" totalImagesPullingTime="3.365101658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:59.223896479 +0000 UTC m=+514.614433823" watchObservedRunningTime="2026-09-11 12:22:59.224270076 +0000 UTC m=+514.614807417"
Sep 11 12:22:59.225637 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:22:59.225594    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn" podStartSLOduration=5.2255793090000004 podStartE2EDuration="5.225579309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:56.204752613 +0000 UTC m=+511.595289955" watchObservedRunningTime="2026-09-11 12:22:59.225579309 +0000 UTC m=+514.616116651"
Sep 11 12:23:01.601468 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:01.601427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:01.701721 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:01.701686    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:06.800281 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:06.800241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:06.902526 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:06.902496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:07.190907 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:07.190877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vq5jn"
Sep 11 12:23:07.212804 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:07.212750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:07.408674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:07.408639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:07.506688 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:07.506615    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:10.207972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:10.207938    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:23:13.865882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:13.865842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:13.962806 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:13.962752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:18.620586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:18.620501    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:18.721431 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:18.721380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:22.632727 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.632688    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"]
Sep 11 12:23:22.633234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.632975    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq" podUID="c894c769-7958-44a3-b384-a2400135da45" containerName="manager" containerID="cri-o://a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459" gracePeriod=10
Sep 11 12:23:22.875701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.875675    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:23:22.930969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.930901    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twg4p\" (UniqueName: \"kubernetes.io/projected/c894c769-7958-44a3-b384-a2400135da45-kube-api-access-twg4p\") pod \"c894c769-7958-44a3-b384-a2400135da45\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") "
Sep 11 12:23:22.930969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.930942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c894c769-7958-44a3-b384-a2400135da45-extensions-socket-volume\") pod \"c894c769-7958-44a3-b384-a2400135da45\" (UID: \"c894c769-7958-44a3-b384-a2400135da45\") "
Sep 11 12:23:22.931473 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.931451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c894c769-7958-44a3-b384-a2400135da45-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c894c769-7958-44a3-b384-a2400135da45" (UID: "c894c769-7958-44a3-b384-a2400135da45"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:23:22.932979 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:22.932961    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c894c769-7958-44a3-b384-a2400135da45-kube-api-access-twg4p" (OuterVolumeSpecName: "kube-api-access-twg4p") pod "c894c769-7958-44a3-b384-a2400135da45" (UID: "c894c769-7958-44a3-b384-a2400135da45"). InnerVolumeSpecName "kube-api-access-twg4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:23:23.032335 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.032307    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twg4p\" (UniqueName: \"kubernetes.io/projected/c894c769-7958-44a3-b384-a2400135da45-kube-api-access-twg4p\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:23:23.032335 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.032330    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c894c769-7958-44a3-b384-a2400135da45-extensions-socket-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:23:23.306547 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.306450    2561 generic.go:356] "Generic (PLEG): container finished" podID="c894c769-7958-44a3-b384-a2400135da45" containerID="a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459" exitCode=0
Sep 11 12:23:23.306547 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.306514    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"
Sep 11 12:23:23.306547 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.306534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq" event={"ID":"c894c769-7958-44a3-b384-a2400135da45","Type":"ContainerDied","Data":"a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459"}
Sep 11 12:23:23.306801 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.306620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq" event={"ID":"c894c769-7958-44a3-b384-a2400135da45","Type":"ContainerDied","Data":"0f06497cdbcf18a9d03e0091f211330a3492c479cc5112087995d882a5cf9004"}
Sep 11 12:23:23.306801 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.306644    2561 scope.go:117] "RemoveContainer" containerID="a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459"
Sep 11 12:23:23.314676 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.314647    2561 scope.go:117] "RemoveContainer" containerID="a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459"
Sep 11 12:23:23.314956 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:23:23.314935    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459\": container with ID starting with a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459 not found: ID does not exist" containerID="a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459"
Sep 11 12:23:23.315024 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.314966    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459"} err="failed to get container status \"a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459\": rpc error: code = NotFound desc = could not find container \"a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459\": container with ID starting with a11f9d53a7836186fadb4ee9867a16dcbf4740affd92f1ea58c8d6b897d7e459 not found: ID does not exist"
Sep 11 12:23:23.332537 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.332510    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"]
Sep 11 12:23:23.337124 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.337101    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-fvkqq"]
Sep 11 12:23:23.477808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.477762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:23.563821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:23.563729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:25.105669 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:25.105644    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c894c769-7958-44a3-b384-a2400135da45" path="/var/lib/kubelet/pods/c894c769-7958-44a3-b384-a2400135da45/volumes"
Sep 11 12:23:26.727352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:26.727320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:26.792271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:26.792237    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:41.132738 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.132695    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:41.133198 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.133061    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c894c769-7958-44a3-b384-a2400135da45" containerName="manager"
Sep 11 12:23:41.133198 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.133073    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c894c769-7958-44a3-b384-a2400135da45" containerName="manager"
Sep 11 12:23:41.133198 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.133126    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c894c769-7958-44a3-b384-a2400135da45" containerName="manager"
Sep 11 12:23:41.136095 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.136074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.138618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.138598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tvpw6\""
Sep 11 12:23:41.138724 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.138648    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 12:23:41.144217 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.144187    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:41.230282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.230252    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:41.259340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.259311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:41.284526 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.284495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3625a1e-f191-4674-8573-415d10b1c818-config-file\") pod \"limitador-limitador-8588749977-65l29\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") " pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.284526 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.284526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6hl\" (UniqueName: \"kubernetes.io/projected/e3625a1e-f191-4674-8573-415d10b1c818-kube-api-access-8p6hl\") pod \"limitador-limitador-8588749977-65l29\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") " pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.385063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.384989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3625a1e-f191-4674-8573-415d10b1c818-config-file\") pod \"limitador-limitador-8588749977-65l29\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") " pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.385063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.385018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6hl\" (UniqueName: \"kubernetes.io/projected/e3625a1e-f191-4674-8573-415d10b1c818-kube-api-access-8p6hl\") pod \"limitador-limitador-8588749977-65l29\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") " pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.385605 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.385588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3625a1e-f191-4674-8573-415d10b1c818-config-file\") pod \"limitador-limitador-8588749977-65l29\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") " pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.394502 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.394476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6hl\" (UniqueName: \"kubernetes.io/projected/e3625a1e-f191-4674-8573-415d10b1c818-kube-api-access-8p6hl\") pod \"limitador-limitador-8588749977-65l29\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") " pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.446109 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.446083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:41.714303 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.714276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:41.716094 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:23:41.716067    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3625a1e_f191_4674_8573_415d10b1c818.slice/crio-3300cb21be711c104d7fe4757d10894d0fae5f2be0f51ba76cb64efb60ec6af0 WatchSource:0}: Error finding container 3300cb21be711c104d7fe4757d10894d0fae5f2be0f51ba76cb64efb60ec6af0: Status 404 returned error can't find the container with id 3300cb21be711c104d7fe4757d10894d0fae5f2be0f51ba76cb64efb60ec6af0
Sep 11 12:23:41.779843 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.779809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:41.973995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:41.973916    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:42.073813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:42.073782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:42.388448 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:42.388417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-65l29" event={"ID":"e3625a1e-f191-4674-8573-415d10b1c818","Type":"ContainerStarted","Data":"3300cb21be711c104d7fe4757d10894d0fae5f2be0f51ba76cb64efb60ec6af0"}
Sep 11 12:23:43.395082 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:43.395047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-65l29" event={"ID":"e3625a1e-f191-4674-8573-415d10b1c818","Type":"ContainerStarted","Data":"153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f"}
Sep 11 12:23:43.395517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:43.395182    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:43.414839 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:43.414792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-65l29" podStartSLOduration=1.337520614 podStartE2EDuration="2.414777048s" totalImagesPullingTime="1.077256434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:43.412766273 +0000 UTC m=+558.803303617" watchObservedRunningTime="2026-09-11 12:23:43.414777048 +0000 UTC m=+558.805314390"
Sep 11 12:23:47.508999 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:47.508964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:47.567517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:47.567477    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:48.860383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:48.860347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:48.931722 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:48.931690    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:23:54.401314 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:54.401276    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:56.208294 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.208261    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:56.208693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.208475    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-65l29" podUID="e3625a1e-f191-4674-8573-415d10b1c818" containerName="limitador" containerID="cri-o://153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f" gracePeriod=30
Sep 11 12:23:56.748160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.748131    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:56.917308 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.917277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3625a1e-f191-4674-8573-415d10b1c818-config-file\") pod \"e3625a1e-f191-4674-8573-415d10b1c818\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") "
Sep 11 12:23:56.917568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.917328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p6hl\" (UniqueName: \"kubernetes.io/projected/e3625a1e-f191-4674-8573-415d10b1c818-kube-api-access-8p6hl\") pod \"e3625a1e-f191-4674-8573-415d10b1c818\" (UID: \"e3625a1e-f191-4674-8573-415d10b1c818\") "
Sep 11 12:23:56.917666 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.917644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3625a1e-f191-4674-8573-415d10b1c818-config-file" (OuterVolumeSpecName: "config-file") pod "e3625a1e-f191-4674-8573-415d10b1c818" (UID: "e3625a1e-f191-4674-8573-415d10b1c818"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:23:56.919430 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:56.919386    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3625a1e-f191-4674-8573-415d10b1c818-kube-api-access-8p6hl" (OuterVolumeSpecName: "kube-api-access-8p6hl") pod "e3625a1e-f191-4674-8573-415d10b1c818" (UID: "e3625a1e-f191-4674-8573-415d10b1c818"). InnerVolumeSpecName "kube-api-access-8p6hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:23:57.018189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.018164    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e3625a1e-f191-4674-8573-415d10b1c818-config-file\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:23:57.018189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.018187    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p6hl\" (UniqueName: \"kubernetes.io/projected/e3625a1e-f191-4674-8573-415d10b1c818-kube-api-access-8p6hl\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:23:57.458036 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.458006    2561 generic.go:356] "Generic (PLEG): container finished" podID="e3625a1e-f191-4674-8573-415d10b1c818" containerID="153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f" exitCode=0
Sep 11 12:23:57.458524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.458049    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-65l29"
Sep 11 12:23:57.458524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.458091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-65l29" event={"ID":"e3625a1e-f191-4674-8573-415d10b1c818","Type":"ContainerDied","Data":"153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f"}
Sep 11 12:23:57.458524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.458155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-65l29" event={"ID":"e3625a1e-f191-4674-8573-415d10b1c818","Type":"ContainerDied","Data":"3300cb21be711c104d7fe4757d10894d0fae5f2be0f51ba76cb64efb60ec6af0"}
Sep 11 12:23:57.458524 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.458190    2561 scope.go:117] "RemoveContainer" containerID="153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f"
Sep 11 12:23:57.466301 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.466264    2561 scope.go:117] "RemoveContainer" containerID="153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f"
Sep 11 12:23:57.466567 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:23:57.466549    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f\": container with ID starting with 153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f not found: ID does not exist" containerID="153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f"
Sep 11 12:23:57.466632 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.466574    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f"} err="failed to get container status \"153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f\": rpc error: code = NotFound desc = could not find container \"153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f\": container with ID starting with 153b2702308353d738dfe4ceffc09d613e147d3817b2440c2ba515721629186f not found: ID does not exist"
Sep 11 12:23:57.474597 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.474531    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:57.475891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:57.475873    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-65l29"]
Sep 11 12:23:59.109194 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:23:59.109147    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3625a1e-f191-4674-8573-415d10b1c818" path="/var/lib/kubelet/pods/e3625a1e-f191-4674-8573-415d10b1c818/volumes"
Sep 11 12:24:00.612318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:00.612287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:00.711844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:00.711810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:01.114367 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.114336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"]
Sep 11 12:24:01.114839 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.114823    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3625a1e-f191-4674-8573-415d10b1c818" containerName="limitador"
Sep 11 12:24:01.114900 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.114844    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3625a1e-f191-4674-8573-415d10b1c818" containerName="limitador"
Sep 11 12:24:01.114957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.114946    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3625a1e-f191-4674-8573-415d10b1c818" containerName="limitador"
Sep 11 12:24:01.118241 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.118226    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.120752 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.120729    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 12:24:01.121736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.121716    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 12:24:01.121829 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.121744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:24:01.121829 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.121761    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cs7fv\""
Sep 11 12:24:01.124607 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.124584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"]
Sep 11 12:24:01.144582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.144558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:01.153347 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.153321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4df10631-9cd8-4c01-9e26-0b5dadb94ac7-data\") pod \"postgres-69fb4594cf-zz4sn\" (UID: \"4df10631-9cd8-4c01-9e26-0b5dadb94ac7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.153492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.153441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krf25\" (UniqueName: \"kubernetes.io/projected/4df10631-9cd8-4c01-9e26-0b5dadb94ac7-kube-api-access-krf25\") pod \"postgres-69fb4594cf-zz4sn\" (UID: \"4df10631-9cd8-4c01-9e26-0b5dadb94ac7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.254584 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.254554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4df10631-9cd8-4c01-9e26-0b5dadb94ac7-data\") pod \"postgres-69fb4594cf-zz4sn\" (UID: \"4df10631-9cd8-4c01-9e26-0b5dadb94ac7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.254726 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.254610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krf25\" (UniqueName: \"kubernetes.io/projected/4df10631-9cd8-4c01-9e26-0b5dadb94ac7-kube-api-access-krf25\") pod \"postgres-69fb4594cf-zz4sn\" (UID: \"4df10631-9cd8-4c01-9e26-0b5dadb94ac7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.255024 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.255004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4df10631-9cd8-4c01-9e26-0b5dadb94ac7-data\") pod \"postgres-69fb4594cf-zz4sn\" (UID: \"4df10631-9cd8-4c01-9e26-0b5dadb94ac7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.262628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.262607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krf25\" (UniqueName: \"kubernetes.io/projected/4df10631-9cd8-4c01-9e26-0b5dadb94ac7-kube-api-access-krf25\") pod \"postgres-69fb4594cf-zz4sn\" (UID: \"4df10631-9cd8-4c01-9e26-0b5dadb94ac7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.430985 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.430904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:01.699560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:01.699534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"]
Sep 11 12:24:01.702046 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:24:01.702016    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df10631_9cd8_4c01_9e26_0b5dadb94ac7.slice/crio-7f175796b6b8bc2e7220d88e92a24bd1e37504495a6ce6f8ac4ae94be0945d81 WatchSource:0}: Error finding container 7f175796b6b8bc2e7220d88e92a24bd1e37504495a6ce6f8ac4ae94be0945d81: Status 404 returned error can't find the container with id 7f175796b6b8bc2e7220d88e92a24bd1e37504495a6ce6f8ac4ae94be0945d81
Sep 11 12:24:02.488086 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:02.488050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn" event={"ID":"4df10631-9cd8-4c01-9e26-0b5dadb94ac7","Type":"ContainerStarted","Data":"7f175796b6b8bc2e7220d88e92a24bd1e37504495a6ce6f8ac4ae94be0945d81"}
Sep 11 12:24:06.509538 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:06.509438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn" event={"ID":"4df10631-9cd8-4c01-9e26-0b5dadb94ac7","Type":"ContainerStarted","Data":"47636e13e4f7e25e1808e7f0c4f67ea40f1fbedd6c6b5c39957fc70baebb6578"}
Sep 11 12:24:06.509910 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:06.509551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:06.529837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:06.529793    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn" podStartSLOduration=1.066195325 podStartE2EDuration="5.529780364s" totalImagesPullingTime="4.463585039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:24:06.529057766 +0000 UTC m=+581.919595102" watchObservedRunningTime="2026-09-11 12:24:06.529780364 +0000 UTC m=+581.920317706"
Sep 11 12:24:12.541926 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:12.541896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zz4sn"
Sep 11 12:24:12.900440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:12.900385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:13.316220 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:13.316120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:21.628669 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.628634    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:21.663890 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.663860    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74cd758b8-kfs9w"]
Sep 11 12:24:21.666698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.666678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.669186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.669155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 12:24:21.669186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.669176    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 12:24:21.670137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.670118    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 12:24:21.670261 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.670223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-72sgl\""
Sep 11 12:24:21.677310 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.677276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cd758b8-kfs9w"]
Sep 11 12:24:21.700583 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.700553    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:21.717612 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.717577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.717720 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.717632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbrx\" (UniqueName: \"kubernetes.io/projected/b5f7b84b-5cec-41f4-b371-760360c96e71-kube-api-access-chbrx\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.717793 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.717770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-webhook-cert\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.760487 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.760458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:21.818254 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.818229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.818390 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.818277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbrx\" (UniqueName: \"kubernetes.io/projected/b5f7b84b-5cec-41f4-b371-760360c96e71-kube-api-access-chbrx\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.818390 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.818360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-webhook-cert\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.818390 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:24:21.818375    2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 12:24:21.818591 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:24:21.818464    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs podName:b5f7b84b-5cec-41f4-b371-760360c96e71 nodeName:}" failed. No retries permitted until 2026-09-11 12:24:22.318445966 +0000 UTC m=+597.708983298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs") pod "maas-controller-74cd758b8-kfs9w" (UID: "b5f7b84b-5cec-41f4-b371-760360c96e71") : secret "maas-controller-metrics-tls" not found
Sep 11 12:24:21.820781 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.820760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-webhook-cert\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.830713 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.830692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbrx\" (UniqueName: \"kubernetes.io/projected/b5f7b84b-5cec-41f4-b371-760360c96e71-kube-api-access-chbrx\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:21.850284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.850259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:21.853982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.853959    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58467f79fd-fp5mk"]
Sep 11 12:24:21.856340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.856322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:21.868271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.868251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58467f79fd-fp5mk"]
Sep 11 12:24:21.910447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.910355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:21.919527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.919506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkgx\" (UniqueName: \"kubernetes.io/projected/3c3916d2-4aac-4a68-ab82-f16995fad2f0-kube-api-access-xnkgx\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:21.919608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.919539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-metrics-certs\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:21.919608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:21.919560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-webhook-cert\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.020133 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.020105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-metrics-certs\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.020249 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.020138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-webhook-cert\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.020249 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.020201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkgx\" (UniqueName: \"kubernetes.io/projected/3c3916d2-4aac-4a68-ab82-f16995fad2f0-kube-api-access-xnkgx\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.022616 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.022593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-webhook-cert\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.022719 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.022649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-metrics-certs\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.031316 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.031287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkgx\" (UniqueName: \"kubernetes.io/projected/3c3916d2-4aac-4a68-ab82-f16995fad2f0-kube-api-access-xnkgx\") pod \"maas-controller-58467f79fd-fp5mk\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") " pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.166134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.166059    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:22.322586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.322553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:22.324676 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.324652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs\") pod \"maas-controller-74cd758b8-kfs9w\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") " pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:22.451115 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.451092    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58467f79fd-fp5mk"]
Sep 11 12:24:22.452227 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:24:22.452203    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3916d2_4aac_4a68_ab82_f16995fad2f0.slice/crio-82438ce70670e89de8b985a9951852876e5247eb849220014a81dde1bbbb112b WatchSource:0}: Error finding container 82438ce70670e89de8b985a9951852876e5247eb849220014a81dde1bbbb112b: Status 404 returned error can't find the container with id 82438ce70670e89de8b985a9951852876e5247eb849220014a81dde1bbbb112b
Sep 11 12:24:22.579297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.579269    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:22.584128 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.584100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-fp5mk" event={"ID":"3c3916d2-4aac-4a68-ab82-f16995fad2f0","Type":"ContainerStarted","Data":"82438ce70670e89de8b985a9951852876e5247eb849220014a81dde1bbbb112b"}
Sep 11 12:24:22.849007 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:22.848979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74cd758b8-kfs9w"]
Sep 11 12:24:22.850654 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:24:22.850627    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f7b84b_5cec_41f4_b371_760360c96e71.slice/crio-821cca1297f8b7707f7c38539f3839c1359cf1fec85cb87fe22dfda3963a5975 WatchSource:0}: Error finding container 821cca1297f8b7707f7c38539f3839c1359cf1fec85cb87fe22dfda3963a5975: Status 404 returned error can't find the container with id 821cca1297f8b7707f7c38539f3839c1359cf1fec85cb87fe22dfda3963a5975
Sep 11 12:24:23.594571 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:23.594530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cd758b8-kfs9w" event={"ID":"b5f7b84b-5cec-41f4-b371-760360c96e71","Type":"ContainerStarted","Data":"821cca1297f8b7707f7c38539f3839c1359cf1fec85cb87fe22dfda3963a5975"}
Sep 11 12:24:25.184340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.184313    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 12:24:25.366683 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.366652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:25.461840 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.461803    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:25.606504 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.606471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cd758b8-kfs9w" event={"ID":"b5f7b84b-5cec-41f4-b371-760360c96e71","Type":"ContainerStarted","Data":"649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d"}
Sep 11 12:24:25.606678 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.606620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:25.608072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.608049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-fp5mk" event={"ID":"3c3916d2-4aac-4a68-ab82-f16995fad2f0","Type":"ContainerStarted","Data":"11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec"}
Sep 11 12:24:25.608188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.608178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:25.628856 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.628767    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74cd758b8-kfs9w" podStartSLOduration=2.291873015 podStartE2EDuration="4.628753998s" totalImagesPullingTime="2.336880983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:24:25.627818341 +0000 UTC m=+601.018355685" watchObservedRunningTime="2026-09-11 12:24:25.628753998 +0000 UTC m=+601.019291340"
Sep 11 12:24:25.659270 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:25.659228    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58467f79fd-fp5mk" podStartSLOduration=1.931511697 podStartE2EDuration="4.659212091s" totalImagesPullingTime="2.727700394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:24:25.657313981 +0000 UTC m=+601.047851324" watchObservedRunningTime="2026-09-11 12:24:25.659212091 +0000 UTC m=+601.049749433"
Sep 11 12:24:36.616414 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:36.616364    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:36.618700 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:36.616571    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:24:36.683131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:36.683095    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74cd758b8-kfs9w"]
Sep 11 12:24:36.683630 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:36.683567    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74cd758b8-kfs9w" podUID="b5f7b84b-5cec-41f4-b371-760360c96e71" containerName="manager" containerID="cri-o://649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d" gracePeriod=30
Sep 11 12:24:36.927263 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:36.927238    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:37.051240 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.051205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chbrx\" (UniqueName: \"kubernetes.io/projected/b5f7b84b-5cec-41f4-b371-760360c96e71-kube-api-access-chbrx\") pod \"b5f7b84b-5cec-41f4-b371-760360c96e71\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") "
Sep 11 12:24:37.051449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.051254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-webhook-cert\") pod \"b5f7b84b-5cec-41f4-b371-760360c96e71\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") "
Sep 11 12:24:37.051449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.051282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs\") pod \"b5f7b84b-5cec-41f4-b371-760360c96e71\" (UID: \"b5f7b84b-5cec-41f4-b371-760360c96e71\") "
Sep 11 12:24:37.053645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.053609    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b5f7b84b-5cec-41f4-b371-760360c96e71" (UID: "b5f7b84b-5cec-41f4-b371-760360c96e71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:24:37.053848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.053824    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f7b84b-5cec-41f4-b371-760360c96e71-kube-api-access-chbrx" (OuterVolumeSpecName: "kube-api-access-chbrx") pod "b5f7b84b-5cec-41f4-b371-760360c96e71" (UID: "b5f7b84b-5cec-41f4-b371-760360c96e71"). InnerVolumeSpecName "kube-api-access-chbrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:24:37.053848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.053830    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b5f7b84b-5cec-41f4-b371-760360c96e71" (UID: "b5f7b84b-5cec-41f4-b371-760360c96e71"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:24:37.152264 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.152234    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chbrx\" (UniqueName: \"kubernetes.io/projected/b5f7b84b-5cec-41f4-b371-760360c96e71-kube-api-access-chbrx\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:24:37.152264 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.152261    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-webhook-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:24:37.152264 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.152271    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5f7b84b-5cec-41f4-b371-760360c96e71-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:24:37.664363 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.664323    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5f7b84b-5cec-41f4-b371-760360c96e71" containerID="649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d" exitCode=0
Sep 11 12:24:37.664854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.664375    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74cd758b8-kfs9w"
Sep 11 12:24:37.664854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.664421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cd758b8-kfs9w" event={"ID":"b5f7b84b-5cec-41f4-b371-760360c96e71","Type":"ContainerDied","Data":"649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d"}
Sep 11 12:24:37.664854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.664469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74cd758b8-kfs9w" event={"ID":"b5f7b84b-5cec-41f4-b371-760360c96e71","Type":"ContainerDied","Data":"821cca1297f8b7707f7c38539f3839c1359cf1fec85cb87fe22dfda3963a5975"}
Sep 11 12:24:37.664854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.664491    2561 scope.go:117] "RemoveContainer" containerID="649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d"
Sep 11 12:24:37.672421 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.672391    2561 scope.go:117] "RemoveContainer" containerID="649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d"
Sep 11 12:24:37.672684 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:24:37.672663    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d\": container with ID starting with 649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d not found: ID does not exist" containerID="649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d"
Sep 11 12:24:37.672734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.672693    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d"} err="failed to get container status \"649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d\": rpc error: code = NotFound desc = could not find container \"649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d\": container with ID starting with 649342eb4033c06232b5bbf553cb097cddb9a32479e094ac0c5a9154209f178d not found: ID does not exist"
Sep 11 12:24:37.690342 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.690317    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74cd758b8-kfs9w"]
Sep 11 12:24:37.695323 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:37.695303    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74cd758b8-kfs9w"]
Sep 11 12:24:39.106069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:39.106033    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f7b84b-5cec-41f4-b371-760360c96e71" path="/var/lib/kubelet/pods/b5f7b84b-5cec-41f4-b371-760360c96e71/volumes"
Sep 11 12:24:39.816750 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:39.816715    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:41.977832 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:41.977796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:57.212610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:57.212570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:57.314078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:57.314043    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:59.024630 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:59.024593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:24:59.125417 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:24:59.125368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:00.516444 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.516392    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:00.655628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.655587    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"]
Sep 11 12:25:00.655947 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.655935    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f7b84b-5cec-41f4-b371-760360c96e71" containerName="manager"
Sep 11 12:25:00.656009 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.655948    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f7b84b-5cec-41f4-b371-760360c96e71" containerName="manager"
Sep 11 12:25:00.656047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.656013    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f7b84b-5cec-41f4-b371-760360c96e71" containerName="manager"
Sep 11 12:25:00.661300 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.661281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.663812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.663769    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 12:25:00.664008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.663945    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:25:00.664096 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.664035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5tx4\""
Sep 11 12:25:00.670106 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.670082    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"]
Sep 11 12:25:00.763168 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.763138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-maas-api-tls\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.763327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.763184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8981219b-7b2e-4505-a669-dbbe1473c69f-openshift-service-ca\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.763327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.763214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsv4\" (UniqueName: \"kubernetes.io/projected/8981219b-7b2e-4505-a669-dbbe1473c69f-kube-api-access-lcsv4\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.763327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.763268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.864062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.864030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-maas-api-tls\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.864062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.864063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8981219b-7b2e-4505-a669-dbbe1473c69f-openshift-service-ca\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.864295 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.864082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsv4\" (UniqueName: \"kubernetes.io/projected/8981219b-7b2e-4505-a669-dbbe1473c69f-kube-api-access-lcsv4\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.864295 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.864183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.864431 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:25:00.864315    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 12:25:00.864431 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:25:00.864427    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs podName:8981219b-7b2e-4505-a669-dbbe1473c69f nodeName:}" failed. No retries permitted until 2026-09-11 12:25:01.36438985 +0000 UTC m=+636.754927184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs") pod "maas-api-655857d65c-cx8jp" (UID: "8981219b-7b2e-4505-a669-dbbe1473c69f") : secret "maas-api-metrics-tls" not found
Sep 11 12:25:00.864809 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.864787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8981219b-7b2e-4505-a669-dbbe1473c69f-openshift-service-ca\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.866697 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.866675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-maas-api-tls\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:00.876955 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:00.876926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsv4\" (UniqueName: \"kubernetes.io/projected/8981219b-7b2e-4505-a669-dbbe1473c69f-kube-api-access-lcsv4\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:01.134554 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.134425    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"]
Sep 11 12:25:01.137731 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.137705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.140153 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.140130    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j6ghs\""
Sep 11 12:25:01.140570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.140552    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 12:25:01.140676 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.140557    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 12:25:01.141012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.140997    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 12:25:01.148604 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.148584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"]
Sep 11 12:25:01.269628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.269590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1446188b-058f-4d03-8064-059b91eedfe7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lb4vt\" (UID: \"1446188b-058f-4d03-8064-059b91eedfe7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.269777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.269665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtvt\" (UniqueName: \"kubernetes.io/projected/1446188b-058f-4d03-8064-059b91eedfe7-kube-api-access-4wtvt\") pod \"payload-pre-processing-59fbc47bf8-lb4vt\" (UID: \"1446188b-058f-4d03-8064-059b91eedfe7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.370658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.370625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1446188b-058f-4d03-8064-059b91eedfe7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lb4vt\" (UID: \"1446188b-058f-4d03-8064-059b91eedfe7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.370820 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.370676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtvt\" (UniqueName: \"kubernetes.io/projected/1446188b-058f-4d03-8064-059b91eedfe7-kube-api-access-4wtvt\") pod \"payload-pre-processing-59fbc47bf8-lb4vt\" (UID: \"1446188b-058f-4d03-8064-059b91eedfe7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.370820 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.370702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:01.371328 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.371307    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1446188b-058f-4d03-8064-059b91eedfe7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lb4vt\" (UID: \"1446188b-058f-4d03-8064-059b91eedfe7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.372948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.372930    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs\") pod \"maas-api-655857d65c-cx8jp\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:01.383001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.382967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtvt\" (UniqueName: \"kubernetes.io/projected/1446188b-058f-4d03-8064-059b91eedfe7-kube-api-access-4wtvt\") pod \"payload-pre-processing-59fbc47bf8-lb4vt\" (UID: \"1446188b-058f-4d03-8064-059b91eedfe7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.448845 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.448773    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:01.572900 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.572865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:01.680908 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.680875    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:01.721065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.721042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"]
Sep 11 12:25:01.722957 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:25:01.722929    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1446188b_058f_4d03_8064_059b91eedfe7.slice/crio-0c5b0e0ba7e24b2503e4fdcfbe0aaf85aa668bdfab80d4ea3d7e2228db62f70a WatchSource:0}: Error finding container 0c5b0e0ba7e24b2503e4fdcfbe0aaf85aa668bdfab80d4ea3d7e2228db62f70a: Status 404 returned error can't find the container with id 0c5b0e0ba7e24b2503e4fdcfbe0aaf85aa668bdfab80d4ea3d7e2228db62f70a
Sep 11 12:25:01.745702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.745670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:01.772764 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.772735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt" event={"ID":"1446188b-058f-4d03-8064-059b91eedfe7","Type":"ContainerStarted","Data":"0c5b0e0ba7e24b2503e4fdcfbe0aaf85aa668bdfab80d4ea3d7e2228db62f70a"}
Sep 11 12:25:01.841616 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:01.841577    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"]
Sep 11 12:25:01.844020 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:25:01.843992    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8981219b_7b2e_4505_a669_dbbe1473c69f.slice/crio-33a751440811160bfeaffb6b2a8710dbae0bcdbfe85750a934374d56025803df WatchSource:0}: Error finding container 33a751440811160bfeaffb6b2a8710dbae0bcdbfe85750a934374d56025803df: Status 404 returned error can't find the container with id 33a751440811160bfeaffb6b2a8710dbae0bcdbfe85750a934374d56025803df
Sep 11 12:25:02.782482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:02.782385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" event={"ID":"8981219b-7b2e-4505-a669-dbbe1473c69f","Type":"ContainerStarted","Data":"33a751440811160bfeaffb6b2a8710dbae0bcdbfe85750a934374d56025803df"}
Sep 11 12:25:04.794011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:04.793968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt" event={"ID":"1446188b-058f-4d03-8064-059b91eedfe7","Type":"ContainerStarted","Data":"f5fe98fa1524ab427bfb06649dc1f1ac15ad3a07fe57212d861d7f46f784fccc"}
Sep 11 12:25:04.794392 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:04.794069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:04.795612 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:04.795589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" event={"ID":"8981219b-7b2e-4505-a669-dbbe1473c69f","Type":"ContainerStarted","Data":"fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92"}
Sep 11 12:25:04.795739 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:04.795723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:04.812326 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:04.812252    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt" podStartSLOduration=0.917836352 podStartE2EDuration="3.812234188s" totalImagesPullingTime="2.894397836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:04.812025442 +0000 UTC m=+640.202562786" watchObservedRunningTime="2026-09-11 12:25:04.812234188 +0000 UTC m=+640.202771531"
Sep 11 12:25:04.834235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:04.834186    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" podStartSLOduration=2.111961321 podStartE2EDuration="4.834171125s" totalImagesPullingTime="2.722209804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:04.832884226 +0000 UTC m=+640.223421568" watchObservedRunningTime="2026-09-11 12:25:04.834171125 +0000 UTC m=+640.224708497"
Sep 11 12:25:10.805689 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:10.805659    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:25:11.767364 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:11.767335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:11.867214 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:11.867179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:15.802092 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:15.802061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lb4vt"
Sep 11 12:25:23.107706 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.107676    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"]
Sep 11 12:25:23.111289 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.111266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.113799 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.113778    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 12:25:23.114450 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.114433    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r6tj8\""
Sep 11 12:25:23.114570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.114451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 12:25:23.114570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.114495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 12:25:23.121678 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.121653    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"]
Sep 11 12:25:23.133861 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.133831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:23.262614 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.262583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.262783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.262633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.262783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.262705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.262783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.262773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.262909 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.262830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbzq\" (UniqueName: \"kubernetes.io/projected/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-kube-api-access-jxbzq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.262950 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.262902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364067 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.363987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbzq\" (UniqueName: \"kubernetes.io/projected/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-kube-api-access-jxbzq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364067 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364067 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.364717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.364661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.366497 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.366476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.366650 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.366632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.370768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.370750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbzq\" (UniqueName: \"kubernetes.io/projected/1c425ab8-bbac-4bc9-ac77-419f4e5c10f3-kube-api-access-jxbzq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64\" (UID: \"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.422802 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.422772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:23.700775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.700749    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"]
Sep 11 12:25:23.703174 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:25:23.703135    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c425ab8_bbac_4bc9_ac77_419f4e5c10f3.slice/crio-c2b12feb193a38b43f2a85f92b3bea3fd6728c264931fc19200c5f1243b0ee5a WatchSource:0}: Error finding container c2b12feb193a38b43f2a85f92b3bea3fd6728c264931fc19200c5f1243b0ee5a: Status 404 returned error can't find the container with id c2b12feb193a38b43f2a85f92b3bea3fd6728c264931fc19200c5f1243b0ee5a
Sep 11 12:25:23.704702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.704679    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:25:23.883770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:23.883740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64" event={"ID":"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3","Type":"ContainerStarted","Data":"c2b12feb193a38b43f2a85f92b3bea3fd6728c264931fc19200c5f1243b0ee5a"}
Sep 11 12:25:25.897276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:25.897236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64" event={"ID":"1c425ab8-bbac-4bc9-ac77-419f4e5c10f3","Type":"ContainerStarted","Data":"a6def6bc395dbfb3484385b18d3ad39389a93150f36a4e864bd5a61237065e36"}
Sep 11 12:25:28.192276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.192214    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64" podStartSLOduration=4.080932854 podStartE2EDuration="5.192197278s" totalImagesPullingTime="1.111264424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:25.915722215 +0000 UTC m=+661.306259557" watchObservedRunningTime="2026-09-11 12:25:28.192197278 +0000 UTC m=+663.582734620"
Sep 11 12:25:28.193875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.193850    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"]
Sep 11 12:25:28.198456 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.198436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.200924 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.200895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 12:25:28.207712 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.207689    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"]
Sep 11 12:25:28.230506 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.230472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:28.307365 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.307335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.307542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.307425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.307542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.307479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.307542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.307522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.307542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.307541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92577119-65c3-41bf-b42e-91a5c174ef1a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.307710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.307561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphzr\" (UniqueName: \"kubernetes.io/projected/92577119-65c3-41bf-b42e-91a5c174ef1a-kube-api-access-qphzr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.408701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.408666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.408850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.408811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.408850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.408835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.408933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.408857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.408933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.408877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92577119-65c3-41bf-b42e-91a5c174ef1a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.408933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.408894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphzr\" (UniqueName: \"kubernetes.io/projected/92577119-65c3-41bf-b42e-91a5c174ef1a-kube-api-access-qphzr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.409208 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.409184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.409287 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.409240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.409287 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.409275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.410938 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.410919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92577119-65c3-41bf-b42e-91a5c174ef1a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.411293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.411271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92577119-65c3-41bf-b42e-91a5c174ef1a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.417188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.417166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphzr\" (UniqueName: \"kubernetes.io/projected/92577119-65c3-41bf-b42e-91a5c174ef1a-kube-api-access-qphzr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fd7mf\" (UID: \"92577119-65c3-41bf-b42e-91a5c174ef1a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.510369 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.510298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:28.781063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.781038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"]
Sep 11 12:25:28.782274 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:25:28.782245    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92577119_65c3_41bf_b42e_91a5c174ef1a.slice/crio-01ce1a8d19902cc91a498ca6a3c4ad2bca4745de32e5a69137fc92b65647e975 WatchSource:0}: Error finding container 01ce1a8d19902cc91a498ca6a3c4ad2bca4745de32e5a69137fc92b65647e975: Status 404 returned error can't find the container with id 01ce1a8d19902cc91a498ca6a3c4ad2bca4745de32e5a69137fc92b65647e975
Sep 11 12:25:28.911457 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:28.911422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf" event={"ID":"92577119-65c3-41bf-b42e-91a5c174ef1a","Type":"ContainerStarted","Data":"01ce1a8d19902cc91a498ca6a3c4ad2bca4745de32e5a69137fc92b65647e975"}
Sep 11 12:25:29.917705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:29.917670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf" event={"ID":"92577119-65c3-41bf-b42e-91a5c174ef1a","Type":"ContainerStarted","Data":"ccd9e5555a5dcf0c0ca8a1088b3e54936f6117599bd79cf26516b0aa7c6e14b7"}
Sep 11 12:25:29.936942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:29.936891    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf" podStartSLOduration=1.71020176 podStartE2EDuration="1.93687313s" totalImagesPullingTime="226.67137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:29.935827519 +0000 UTC m=+665.326364862" watchObservedRunningTime="2026-09-11 12:25:29.93687313 +0000 UTC m=+665.327410471"
Sep 11 12:25:33.423716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:33.423681    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:33.424112 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:33.423766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:33.436773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:33.436748    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:33.804819 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:33.804718    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:33.946628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:33.946594    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64"
Sep 11 12:25:38.511250 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:38.511213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:38.511653 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:38.511285    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:38.523668 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:38.523643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:38.969168 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:38.969139    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fd7mf"
Sep 11 12:25:51.030263 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:51.030227    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:25:58.237324 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:25:58.237292    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:26:01.038309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:26:01.038271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:26:20.624306 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:26:20.624223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:26:30.642727 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:26:30.642694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:26:51.700637 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:26:51.700587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:39.733460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:39.733419    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:39.832546 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:39.832517    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:40.843878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:40.843844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:40.948847 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:40.948806    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.203526 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.203486    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.269584 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.269551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.344299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.344258    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.518672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.518583    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.582578 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.582520    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"]
Sep 11 12:27:56.586068 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.586045    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.589884 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.589863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 12:27:56.589995 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.589937    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cb20c8-openshift-default-dockercfg-kgkh9\""
Sep 11 12:27:56.602973 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.602951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"]
Sep 11 12:27:56.620393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.620363    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.703800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-envoy\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.703952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqdn\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-kube-api-access-zvqdn\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.703952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-token\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.703952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-socket\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.703952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-data\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.704134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-certs\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.704134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.703982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-podinfo\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.704134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.704066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istiod-ca-cert\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.704245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.704147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-credential-socket\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.713219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.713187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.787638 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.787562    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.804773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqdn\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-kube-api-access-zvqdn\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.804773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-token\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.804997 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-socket\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.804997 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-data\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.804997 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-certs\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.804997 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-podinfo\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.804997 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.804933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istiod-ca-cert\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805265 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-credential-socket\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805265 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-envoy\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805265 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-socket\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805445 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-certs\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805445 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-data\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-credential-socket\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.805751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.805723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istiod-ca-cert\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.807276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.807255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-envoy\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.807444 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.807425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-podinfo\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.812978 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.812961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-token\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.813345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.813324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqdn\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-kube-api-access-zvqdn\") pod \"e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") " pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:56.887520 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.887476    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:56.897435 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:56.897389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:27:57.176287 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:57.176264    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"]
Sep 11 12:27:57.178243 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:27:57.178217    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0ede1b_c837_4365_a8ca_b21a5e95bab2.slice/crio-ab0d4e52b9b9684f79cd1ae36815c82c9e8247fafc1ca0aca0729b11370d5ff8 WatchSource:0}: Error finding container ab0d4e52b9b9684f79cd1ae36815c82c9e8247fafc1ca0aca0729b11370d5ff8: Status 404 returned error can't find the container with id ab0d4e52b9b9684f79cd1ae36815c82c9e8247fafc1ca0aca0729b11370d5ff8
Sep 11 12:27:57.595984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:57.595947    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:57.601043 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:57.601017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" event={"ID":"3f0ede1b-c837-4365-a8ca-b21a5e95bab2","Type":"ContainerStarted","Data":"ab0d4e52b9b9684f79cd1ae36815c82c9e8247fafc1ca0aca0729b11370d5ff8"}
Sep 11 12:27:57.696983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:57.696942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:58.047333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:58.047239    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:27:59.990190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:27:59.990151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:00.094739 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.094672    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:00.190201 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.190157    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:00.293862 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.293824    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:28:00.293942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.293900    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:28:00.293942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.293930    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:28:00.619734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.619694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" event={"ID":"3f0ede1b-c837-4365-a8ca-b21a5e95bab2","Type":"ContainerStarted","Data":"9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"}
Sep 11 12:28:00.678808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.678748    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podStartSLOduration=1.565652459 podStartE2EDuration="4.678730264s" totalImagesPullingTime="3.113077805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:00.677151747 +0000 UTC m=+816.067689091" watchObservedRunningTime="2026-09-11 12:28:00.678730264 +0000 UTC m=+816.069267605"
Sep 11 12:28:00.898483 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:00.898375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:01.119613 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.119578    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"]
Sep 11 12:28:01.123209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.123188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.125839 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.125817    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1a8ee3-serving-cert\""
Sep 11 12:28:01.133253 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.133228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"]
Sep 11 12:28:01.139930 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.139892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxm5d\" (UniqueName: \"kubernetes.io/projected/80f14a47-9fdc-49c9-b47f-d79105195a32-kube-api-access-bxm5d\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.140063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.139995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.140131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.140091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.140189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.140174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f14a47-9fdc-49c9-b47f-d79105195a32-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.141341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.141311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:01.224346 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.224246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:01.240959 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.240923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f14a47-9fdc-49c9-b47f-d79105195a32-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.241135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.240982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxm5d\" (UniqueName: \"kubernetes.io/projected/80f14a47-9fdc-49c9-b47f-d79105195a32-kube-api-access-bxm5d\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.241135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.241044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.241135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.241126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.241295 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:01.241246    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-serving-cert: secret "maas-api-e2e-worker-w0-1a8ee3-serving-cert" not found
Sep 11 12:28:01.241355 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:01.241303    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls podName:80f14a47-9fdc-49c9-b47f-d79105195a32 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:01.741284906 +0000 UTC m=+817.131822227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls") pod "maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" (UID: "80f14a47-9fdc-49c9-b47f-d79105195a32") : secret "maas-api-e2e-worker-w0-1a8ee3-serving-cert" not found
Sep 11 12:28:01.241687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.241667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f14a47-9fdc-49c9-b47f-d79105195a32-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.243933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.243902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.256189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.256168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxm5d\" (UniqueName: \"kubernetes.io/projected/80f14a47-9fdc-49c9-b47f-d79105195a32-kube-api-access-bxm5d\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.495439 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.495324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:01.659432 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.659351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:01.742820 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.742777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:01.745197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.745162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.747065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.747010    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"]
Sep 11 12:28:01.747659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.747639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:01.750549 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.750532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:01.752790 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.752769    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1a8ee3-dockercfg-zlnlg\""
Sep 11 12:28:01.752909 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.752827    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1a8ee3\""
Sep 11 12:28:01.760786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.760690    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"]
Sep 11 12:28:01.779127 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.779102    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"]
Sep 11 12:28:01.782393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.782378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.792715 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.792684    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"]
Sep 11 12:28:01.845850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.845816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e14e285-5c91-43ed-ad46-e83b2e721bb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:01.846008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.845928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmfs\" (UniqueName: \"kubernetes.io/projected/5e14e285-5c91-43ed-ad46-e83b2e721bb4-kube-api-access-9zmfs\") pod \"payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:01.846008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.845959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nb8\" (UniqueName: \"kubernetes.io/projected/4c6a4ec2-a307-48e9-82ce-c294421b1487-kube-api-access-66nb8\") pod \"payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.846008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.845994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6a4ec2-a307-48e9-82ce-c294421b1487-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.903035 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.903005    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:01.946574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.946538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nb8\" (UniqueName: \"kubernetes.io/projected/4c6a4ec2-a307-48e9-82ce-c294421b1487-kube-api-access-66nb8\") pod \"payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.946574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.946578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6a4ec2-a307-48e9-82ce-c294421b1487-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.946799 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.946616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e14e285-5c91-43ed-ad46-e83b2e721bb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:01.946799 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.946679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmfs\" (UniqueName: \"kubernetes.io/projected/5e14e285-5c91-43ed-ad46-e83b2e721bb4-kube-api-access-9zmfs\") pod \"payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:01.947372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.947347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e14e285-5c91-43ed-ad46-e83b2e721bb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:01.947516 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.947394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6a4ec2-a307-48e9-82ce-c294421b1487-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.954511 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.954488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nb8\" (UniqueName: \"kubernetes.io/projected/4c6a4ec2-a307-48e9-82ce-c294421b1487-kube-api-access-66nb8\") pod \"payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:01.954894 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:01.954874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmfs\" (UniqueName: \"kubernetes.io/projected/5e14e285-5c91-43ed-ad46-e83b2e721bb4-kube-api-access-9zmfs\") pod \"payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:02.035049 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.034969    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:02.066139 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.066104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:02.093776 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.093388    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:02.202584 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.202550    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"]
Sep 11 12:28:02.207830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.207454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.209906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.209879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-38a724-serving-cert\""
Sep 11 12:28:02.219451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.219423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"]
Sep 11 12:28:02.242031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.241772    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:02.250599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.250566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d901b4a-ee48-41a7-be49-52a9f05b5c21-openshift-service-ca\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.250757 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.250675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-metrics-certs\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.250817 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.250782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.250865 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.250829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvvl\" (UniqueName: \"kubernetes.io/projected/4d901b4a-ee48-41a7-be49-52a9f05b5c21-kube-api-access-rqvvl\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.305562 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.305464    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:02.317416 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.317372    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"]
Sep 11 12:28:02.319300 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:02.319278    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f14a47_9fdc_49c9_b47f_d79105195a32.slice/crio-913da58caf2e854aed1eed5a0754be59dd326abe0bfc284408330846f142576c WatchSource:0}: Error finding container 913da58caf2e854aed1eed5a0754be59dd326abe0bfc284408330846f142576c: Status 404 returned error can't find the container with id 913da58caf2e854aed1eed5a0754be59dd326abe0bfc284408330846f142576c
Sep 11 12:28:02.351786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.351517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.351786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.351589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvvl\" (UniqueName: \"kubernetes.io/projected/4d901b4a-ee48-41a7-be49-52a9f05b5c21-kube-api-access-rqvvl\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.351786 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:02.351722    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-serving-cert: secret "maas-api-e2e-inf-a-38a724-serving-cert" not found
Sep 11 12:28:02.352091 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:02.351830    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls podName:4d901b4a-ee48-41a7-be49-52a9f05b5c21 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:02.851792566 +0000 UTC m=+818.242329885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls") pod "maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" (UID: "4d901b4a-ee48-41a7-be49-52a9f05b5c21") : secret "maas-api-e2e-inf-a-38a724-serving-cert" not found
Sep 11 12:28:02.352091 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.352020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d901b4a-ee48-41a7-be49-52a9f05b5c21-openshift-service-ca\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.352211 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.352107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-metrics-certs\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.353322 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.353293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d901b4a-ee48-41a7-be49-52a9f05b5c21-openshift-service-ca\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.354761 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.354741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"]
Sep 11 12:28:02.355325 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.355267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-metrics-certs\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.356946 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:02.356924    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e14e285_5c91_43ed_ad46_e83b2e721bb4.slice/crio-1bebb6d8ed68a29704374bdd8bc81f7b38df032044196586c75480dc4dd0b1e4 WatchSource:0}: Error finding container 1bebb6d8ed68a29704374bdd8bc81f7b38df032044196586c75480dc4dd0b1e4: Status 404 returned error can't find the container with id 1bebb6d8ed68a29704374bdd8bc81f7b38df032044196586c75480dc4dd0b1e4
Sep 11 12:28:02.360777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.360758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvvl\" (UniqueName: \"kubernetes.io/projected/4d901b4a-ee48-41a7-be49-52a9f05b5c21-kube-api-access-rqvvl\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.390578 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.390554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"]
Sep 11 12:28:02.392481 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:02.392452    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6a4ec2_a307_48e9_82ce_c294421b1487.slice/crio-1c707dd2b26c92b9010647e1ed2102b3e093cb8695553350333ce70f3ebd77d8 WatchSource:0}: Error finding container 1c707dd2b26c92b9010647e1ed2102b3e093cb8695553350333ce70f3ebd77d8: Status 404 returned error can't find the container with id 1c707dd2b26c92b9010647e1ed2102b3e093cb8695553350333ce70f3ebd77d8
Sep 11 12:28:02.631138 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.631099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" event={"ID":"80f14a47-9fdc-49c9-b47f-d79105195a32","Type":"ContainerStarted","Data":"913da58caf2e854aed1eed5a0754be59dd326abe0bfc284408330846f142576c"}
Sep 11 12:28:02.632815 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.632756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:02.632955 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.632938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw" event={"ID":"5e14e285-5c91-43ed-ad46-e83b2e721bb4","Type":"ContainerStarted","Data":"756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591"}
Sep 11 12:28:02.633023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.632990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:02.633023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.633014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw" event={"ID":"5e14e285-5c91-43ed-ad46-e83b2e721bb4","Type":"ContainerStarted","Data":"1bebb6d8ed68a29704374bdd8bc81f7b38df032044196586c75480dc4dd0b1e4"}
Sep 11 12:28:02.634542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.634516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk" event={"ID":"4c6a4ec2-a307-48e9-82ce-c294421b1487","Type":"ContainerStarted","Data":"54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5"}
Sep 11 12:28:02.634673 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.634569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk" event={"ID":"4c6a4ec2-a307-48e9-82ce-c294421b1487","Type":"ContainerStarted","Data":"1c707dd2b26c92b9010647e1ed2102b3e093cb8695553350333ce70f3ebd77d8"}
Sep 11 12:28:02.634775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.634757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:02.645526 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.645495    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:02.658576 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.658521    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw" podStartSLOduration=1.6585021599999998 podStartE2EDuration="1.65850216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:02.655276277 +0000 UTC m=+818.045813619" watchObservedRunningTime="2026-09-11 12:28:02.65850216 +0000 UTC m=+818.049039504"
Sep 11 12:28:02.674816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.674764    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk" podStartSLOduration=1.674745953 podStartE2EDuration="1.674745953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:02.674522752 +0000 UTC m=+818.065060093" watchObservedRunningTime="2026-09-11 12:28:02.674745953 +0000 UTC m=+818.065283299"
Sep 11 12:28:02.729156 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.729120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:02.857865 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.857825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:02.858039 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:02.858006    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-serving-cert: secret "maas-api-e2e-inf-a-38a724-serving-cert" not found
Sep 11 12:28:02.858101 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:02.858089    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls podName:4d901b4a-ee48-41a7-be49-52a9f05b5c21 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:03.858068151 +0000 UTC m=+819.248605473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls") pod "maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" (UID: "4d901b4a-ee48-41a7-be49-52a9f05b5c21") : secret "maas-api-e2e-inf-a-38a724-serving-cert" not found
Sep 11 12:28:02.972273 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:02.972242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:03.068504 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.068460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:03.188470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.188384    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"]
Sep 11 12:28:03.192031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.192008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.194435 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.194342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-38a724-dockercfg-9fdfs\""
Sep 11 12:28:03.194435 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.194374    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-38a724\""
Sep 11 12:28:03.203614 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.203590    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"]
Sep 11 12:28:03.262065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.262032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c78d0a-4ed5-4578-ba06-2c85e73edd73-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.262251 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.262145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xfc\" (UniqueName: \"kubernetes.io/projected/58c78d0a-4ed5-4578-ba06-2c85e73edd73-kube-api-access-27xfc\") pod \"payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.362984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.362945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xfc\" (UniqueName: \"kubernetes.io/projected/58c78d0a-4ed5-4578-ba06-2c85e73edd73-kube-api-access-27xfc\") pod \"payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.363172 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.363066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c78d0a-4ed5-4578-ba06-2c85e73edd73-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.363837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.363813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c78d0a-4ed5-4578-ba06-2c85e73edd73-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.375672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.375644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xfc\" (UniqueName: \"kubernetes.io/projected/58c78d0a-4ed5-4578-ba06-2c85e73edd73-kube-api-access-27xfc\") pod \"payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.490285 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.490196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:03.507355 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.507323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:03.641856 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.641789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" event={"ID":"80f14a47-9fdc-49c9-b47f-d79105195a32","Type":"ContainerStarted","Data":"a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba"}
Sep 11 12:28:03.642347 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.641863    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:03.642347 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.642154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:03.642694 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.642677    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:03.660105 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.660064    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" podStartSLOduration=2.06199194 podStartE2EDuration="2.660050235s" totalImagesPullingTime="598.058295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:03.658553868 +0000 UTC m=+819.049091223" watchObservedRunningTime="2026-09-11 12:28:03.660050235 +0000 UTC m=+819.050587576"
Sep 11 12:28:03.790079 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.790054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"]
Sep 11 12:28:03.792601 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:03.792571    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c78d0a_4ed5_4578_ba06_2c85e73edd73.slice/crio-0440d5cb2ef9f3c0876f3642c60a701115f4185bef7d57dcbcc6c1a768501568 WatchSource:0}: Error finding container 0440d5cb2ef9f3c0876f3642c60a701115f4185bef7d57dcbcc6c1a768501568: Status 404 returned error can't find the container with id 0440d5cb2ef9f3c0876f3642c60a701115f4185bef7d57dcbcc6c1a768501568
Sep 11 12:28:03.868646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.868617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:03.871016 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:03.870989    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") pod \"maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:04.021226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.021192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:04.294990 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.294968    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"]
Sep 11 12:28:04.296864 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:04.296838    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d901b4a_ee48_41a7_be49_52a9f05b5c21.slice/crio-1cad0e5fd88e05c8dfc6d8ed99ef30d1583d04a0272bcb5510f6c180a77417d4 WatchSource:0}: Error finding container 1cad0e5fd88e05c8dfc6d8ed99ef30d1583d04a0272bcb5510f6c180a77417d4: Status 404 returned error can't find the container with id 1cad0e5fd88e05c8dfc6d8ed99ef30d1583d04a0272bcb5510f6c180a77417d4
Sep 11 12:28:04.429899 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.429865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:04.528338 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.528300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:04.648601 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.648565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5" event={"ID":"58c78d0a-4ed5-4578-ba06-2c85e73edd73","Type":"ContainerStarted","Data":"f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02"}
Sep 11 12:28:04.648773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.648635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:04.648773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.648662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5" event={"ID":"58c78d0a-4ed5-4578-ba06-2c85e73edd73","Type":"ContainerStarted","Data":"0440d5cb2ef9f3c0876f3642c60a701115f4185bef7d57dcbcc6c1a768501568"}
Sep 11 12:28:04.649730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.649704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" event={"ID":"4d901b4a-ee48-41a7-be49-52a9f05b5c21","Type":"ContainerStarted","Data":"1cad0e5fd88e05c8dfc6d8ed99ef30d1583d04a0272bcb5510f6c180a77417d4"}
Sep 11 12:28:04.667718 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.667671    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5" podStartSLOduration=1.6676540260000001 podStartE2EDuration="1.667654026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:04.666944036 +0000 UTC m=+820.057481378" watchObservedRunningTime="2026-09-11 12:28:04.667654026 +0000 UTC m=+820.058191368"
Sep 11 12:28:04.748996 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.748958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:04.988831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:04.988742    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:05.186470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.186433    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"]
Sep 11 12:28:05.190795 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.190764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.193155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.193131    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-cb20c8-serving-cert\""
Sep 11 12:28:05.201065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.200705    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"]
Sep 11 12:28:05.221678 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.221647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:05.281943 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.281869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.282107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.281956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktcg\" (UniqueName: \"kubernetes.io/projected/70f2e1ae-a69a-4f05-8987-8a66c82acf60-kube-api-access-dktcg\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.282107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.282014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f2e1ae-a69a-4f05-8987-8a66c82acf60-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.282107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.282066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-metrics-certs\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.322682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.322646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:05.383429 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.383379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.383603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.383450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktcg\" (UniqueName: \"kubernetes.io/projected/70f2e1ae-a69a-4f05-8987-8a66c82acf60-kube-api-access-dktcg\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.383603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.383487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f2e1ae-a69a-4f05-8987-8a66c82acf60-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.383603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.383509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-metrics-certs\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.383603 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:05.383513    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-serving-cert: secret "maas-api-e2e-worker-w4-cb20c8-serving-cert" not found
Sep 11 12:28:05.383603 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:05.383585    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls podName:70f2e1ae-a69a-4f05-8987-8a66c82acf60 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:05.883565881 +0000 UTC m=+821.274103210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls") pod "maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" (UID: "70f2e1ae-a69a-4f05-8987-8a66c82acf60") : secret "maas-api-e2e-worker-w4-cb20c8-serving-cert" not found
Sep 11 12:28:05.384198 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.384177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f2e1ae-a69a-4f05-8987-8a66c82acf60-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.386117 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.386095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-metrics-certs\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.393738 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.393706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktcg\" (UniqueName: \"kubernetes.io/projected/70f2e1ae-a69a-4f05-8987-8a66c82acf60-kube-api-access-dktcg\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.521367 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.521317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:05.663651 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.663565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" event={"ID":"4d901b4a-ee48-41a7-be49-52a9f05b5c21","Type":"ContainerStarted","Data":"faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984"}
Sep 11 12:28:05.684263 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.684210    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" podStartSLOduration=3.277508599 podStartE2EDuration="3.684195803s" totalImagesPullingTime="406.687204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:05.682337337 +0000 UTC m=+821.072874679" watchObservedRunningTime="2026-09-11 12:28:05.684195803 +0000 UTC m=+821.074733174"
Sep 11 12:28:05.729284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.729251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:05.827645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.827598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:05.887865 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.887831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.890256 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.890232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:05.928940 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:05.928871    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:06.000254 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.000209    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:06.106013 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.105981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:06.287245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.287165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:06.384130 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.384107    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"]
Sep 11 12:28:06.390755 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:06.390721    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f2e1ae_a69a_4f05_8987_8a66c82acf60.slice/crio-9413779cf4635b78e543cd022d6f4336a4cba4405886a44b58299d6eb03e511b WatchSource:0}: Error finding container 9413779cf4635b78e543cd022d6f4336a4cba4405886a44b58299d6eb03e511b: Status 404 returned error can't find the container with id 9413779cf4635b78e543cd022d6f4336a4cba4405886a44b58299d6eb03e511b
Sep 11 12:28:06.670798 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.670767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" event={"ID":"70f2e1ae-a69a-4f05-8987-8a66c82acf60","Type":"ContainerStarted","Data":"9413779cf4635b78e543cd022d6f4336a4cba4405886a44b58299d6eb03e511b"}
Sep 11 12:28:06.670967 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.670902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:06.720784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:06.720753    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:07.678323 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:07.678276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" event={"ID":"70f2e1ae-a69a-4f05-8987-8a66c82acf60","Type":"ContainerStarted","Data":"1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea"}
Sep 11 12:28:07.678789 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:07.678412    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:07.700149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:07.700096    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" podStartSLOduration=2.366514081 podStartE2EDuration="2.700079042s" totalImagesPullingTime="333.564961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:07.697740424 +0000 UTC m=+823.088277767" watchObservedRunningTime="2026-09-11 12:28:07.700079042 +0000 UTC m=+823.090616387"
Sep 11 12:28:08.128211 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:08.128176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:08.206786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:08.206680    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:08.528944 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:08.528861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:08.662939 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:08.662906    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:09.302011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.301975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:09.363248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.363214    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:09.465160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.465121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:09.655364 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.655338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:28:09.793242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.793207    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:09.890808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.890777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:09.902866 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.902836    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"]
Sep 11 12:28:09.906722 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.906662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:09.909615 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.909584    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9226c5-dockercfg-fdkq7\""
Sep 11 12:28:09.909993 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.909975    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9226c5\""
Sep 11 12:28:09.917582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.917558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"]
Sep 11 12:28:09.928182 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.928157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnmk\" (UniqueName: \"kubernetes.io/projected/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-kube-api-access-lsnmk\") pod \"payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:09.928470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.928434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:09.932659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.932641    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"]
Sep 11 12:28:09.936132 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.936117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:09.945877 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:09.945853    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"]
Sep 11 12:28:10.029714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.029669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnmk\" (UniqueName: \"kubernetes.io/projected/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-kube-api-access-lsnmk\") pod \"payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:10.029714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.029729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:10.029916 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.029803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6148102-bff0-4869-baf8-2482e8d6714e-plugins-config-volume\") pod \"payload-processing-e2e-autores-9226c5-77f966c67-zhb95\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") " pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.029916 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.029835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw275\" (UniqueName: \"kubernetes.io/projected/a6148102-bff0-4869-baf8-2482e8d6714e-kube-api-access-gw275\") pod \"payload-processing-e2e-autores-9226c5-77f966c67-zhb95\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") " pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.030426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.030389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:10.038824 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.038795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnmk\" (UniqueName: \"kubernetes.io/projected/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-kube-api-access-lsnmk\") pod \"payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:10.131183 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.131143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6148102-bff0-4869-baf8-2482e8d6714e-plugins-config-volume\") pod \"payload-processing-e2e-autores-9226c5-77f966c67-zhb95\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") " pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.131416 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.131367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw275\" (UniqueName: \"kubernetes.io/projected/a6148102-bff0-4869-baf8-2482e8d6714e-kube-api-access-gw275\") pod \"payload-processing-e2e-autores-9226c5-77f966c67-zhb95\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") " pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.131887 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.131869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6148102-bff0-4869-baf8-2482e8d6714e-plugins-config-volume\") pod \"payload-processing-e2e-autores-9226c5-77f966c67-zhb95\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") " pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.140930 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.140901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw275\" (UniqueName: \"kubernetes.io/projected/a6148102-bff0-4869-baf8-2482e8d6714e-kube-api-access-gw275\") pod \"payload-processing-e2e-autores-9226c5-77f966c67-zhb95\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") " pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.217823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.217715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:10.248533 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.248500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.320217 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.318493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"]
Sep 11 12:28:10.327324 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.327254    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.329933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.329869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7e8d45b5-serving-cert\""
Sep 11 12:28:10.336439 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.336361    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"]
Sep 11 12:28:10.374470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.374347    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:10.433871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.433834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.434037 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.433914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6dw\" (UniqueName: \"kubernetes.io/projected/feecc3fd-6a6b-42e3-813f-133aab831923-kube-api-access-rz6dw\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.434037 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.433964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feecc3fd-6a6b-42e3-813f-133aab831923-openshift-service-ca\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.434037 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.434011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-metrics-certs\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.469624 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.469539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:10.504914 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.504889    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"]
Sep 11 12:28:10.505844 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:10.505820    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5114de1_a9fb_4d83_b5ac_c4a7646d950d.slice/crio-417633a3925ad2ae205cb81fac31e3513813b561ed393206bb8b4774f9e31bba WatchSource:0}: Error finding container 417633a3925ad2ae205cb81fac31e3513813b561ed393206bb8b4774f9e31bba: Status 404 returned error can't find the container with id 417633a3925ad2ae205cb81fac31e3513813b561ed393206bb8b4774f9e31bba
Sep 11 12:28:10.535149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.534716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.535149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.534792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6dw\" (UniqueName: \"kubernetes.io/projected/feecc3fd-6a6b-42e3-813f-133aab831923-kube-api-access-rz6dw\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.535149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.534840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feecc3fd-6a6b-42e3-813f-133aab831923-openshift-service-ca\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.535149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.534864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-metrics-certs\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.535622 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:10.535585    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-serving-cert: secret "maas-api-e2e-ait-7e8d45b5-serving-cert" not found
Sep 11 12:28:10.535736 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:10.535656    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls podName:feecc3fd-6a6b-42e3-813f-133aab831923 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:11.035638299 +0000 UTC m=+826.426175636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls") pod "maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" (UID: "feecc3fd-6a6b-42e3-813f-133aab831923") : secret "maas-api-e2e-ait-7e8d45b5-serving-cert" not found
Sep 11 12:28:10.535736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.535658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feecc3fd-6a6b-42e3-813f-133aab831923-openshift-service-ca\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.537377 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.537357    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"]
Sep 11 12:28:10.537868 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.537841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-metrics-certs\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.539201 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:10.539178    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6148102_bff0_4869_baf8_2482e8d6714e.slice/crio-2ac3d1bf2834f7af0ca786810f7081b90483d5aacdeb3f015f035eb4fd7533f5 WatchSource:0}: Error finding container 2ac3d1bf2834f7af0ca786810f7081b90483d5aacdeb3f015f035eb4fd7533f5: Status 404 returned error can't find the container with id 2ac3d1bf2834f7af0ca786810f7081b90483d5aacdeb3f015f035eb4fd7533f5
Sep 11 12:28:10.544268 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.544233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6dw\" (UniqueName: \"kubernetes.io/projected/feecc3fd-6a6b-42e3-813f-133aab831923-kube-api-access-rz6dw\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:10.696068 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.696026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95" event={"ID":"a6148102-bff0-4869-baf8-2482e8d6714e","Type":"ContainerStarted","Data":"3b81ae87f4c7fdb95b758228a735bd2ef98127030ab5a7e3fe4e3c46de3be5f9"}
Sep 11 12:28:10.696276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.696093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:10.696276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.696113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95" event={"ID":"a6148102-bff0-4869-baf8-2482e8d6714e","Type":"ContainerStarted","Data":"2ac3d1bf2834f7af0ca786810f7081b90483d5aacdeb3f015f035eb4fd7533f5"}
Sep 11 12:28:10.701194 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.701140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv" event={"ID":"b5114de1-a9fb-4d83-b5ac-c4a7646d950d","Type":"ContainerStarted","Data":"87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9"}
Sep 11 12:28:10.701383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.701197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv" event={"ID":"b5114de1-a9fb-4d83-b5ac-c4a7646d950d","Type":"ContainerStarted","Data":"417633a3925ad2ae205cb81fac31e3513813b561ed393206bb8b4774f9e31bba"}
Sep 11 12:28:10.701383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.701227    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:10.719539 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.719488    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95" podStartSLOduration=1.719470293 podStartE2EDuration="1.719470293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:10.717375791 +0000 UTC m=+826.107913136" watchObservedRunningTime="2026-09-11 12:28:10.719470293 +0000 UTC m=+826.110007635"
Sep 11 12:28:10.736470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:10.736376    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv" podStartSLOduration=1.73636019 podStartE2EDuration="1.73636019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:10.734455286 +0000 UTC m=+826.124992629" watchObservedRunningTime="2026-09-11 12:28:10.73636019 +0000 UTC m=+826.126897532"
Sep 11 12:28:11.042972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:11.042874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:11.046252 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:11.046222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls\") pod \"maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:11.245482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:11.245443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:11.523892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:11.523869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"]
Sep 11 12:28:11.525462 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:11.525435    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeecc3fd_6a6b_42e3_813f_133aab831923.slice/crio-6ec74a6c75c7a51d0c3fda854ba8063eec49a7cbf86844e1ca3d098486ac3adf WatchSource:0}: Error finding container 6ec74a6c75c7a51d0c3fda854ba8063eec49a7cbf86844e1ca3d098486ac3adf: Status 404 returned error can't find the container with id 6ec74a6c75c7a51d0c3fda854ba8063eec49a7cbf86844e1ca3d098486ac3adf
Sep 11 12:28:11.708226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:11.708192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" event={"ID":"feecc3fd-6a6b-42e3-813f-133aab831923","Type":"ContainerStarted","Data":"6ec74a6c75c7a51d0c3fda854ba8063eec49a7cbf86844e1ca3d098486ac3adf"}
Sep 11 12:28:11.984107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:11.984034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:12.039833 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:12.039799    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:12.325044 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:12.325014    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:12.684066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:12.683990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:28:12.716453 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:12.716413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" event={"ID":"feecc3fd-6a6b-42e3-813f-133aab831923","Type":"ContainerStarted","Data":"196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f"}
Sep 11 12:28:12.716618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:12.716559    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:12.734122 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:12.734071    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" podStartSLOduration=2.463738691 podStartE2EDuration="2.734054192s" totalImagesPullingTime="270.315501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:12.733269373 +0000 UTC m=+828.123806715" watchObservedRunningTime="2026-09-11 12:28:12.734054192 +0000 UTC m=+828.124591580"
Sep 11 12:28:13.645619 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:13.645580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:28:13.689471 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:13.689443    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:28:14.652137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:14.652106    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:28:14.820248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:14.820216    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:14.975682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:14.975593    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:15.075293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:15.075197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:15.174429 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:15.174380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:15.665747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:15.665719    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:28:17.937774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:17.937731    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:18.728619 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:18.728591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:19.651509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:19.651472    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:19.744984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:19.744941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:19.902048 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:19.901950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:21.616379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:21.616326    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:21.710505 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:21.710476    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:28:21.710665 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:21.710540    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:28:22.072755 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:22.072677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:23.819816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:23.819779    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:24.962840 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:24.962800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.062747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.062671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.299927 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.299851    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.482878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.482844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.602608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.602571    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.720317 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.720289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.803189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.803160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:25.872374 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:25.872310    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:26.343710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.343673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:26.577807 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.577776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:26.695446 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.695334    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:26.795235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.795205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:26.808164 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.808136    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerID="9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7" exitCode=0
Sep 11 12:28:26.808303 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.808209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" event={"ID":"3f0ede1b-c837-4365-a8ca-b21a5e95bab2","Type":"ContainerDied","Data":"9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"}
Sep 11 12:28:26.808642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.808627    2561 scope.go:117] "RemoveContainer" containerID="9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"
Sep 11 12:28:26.809872 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.809842    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:28:26.809932 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.809914    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:28:26.809969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.809945    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:28:26.889704 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.889670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:26.987618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:26.987525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:27.540988 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:27.540957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:27.814869 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:27.814782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" event={"ID":"3f0ede1b-c837-4365-a8ca-b21a5e95bab2","Type":"ContainerStarted","Data":"5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83"}
Sep 11 12:28:27.897564 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:27.897532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:27.898622 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:27.898594    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 11 12:28:27.898736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:27.898645    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 11 12:28:28.009908 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:28.009864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:28.112095 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:28.112061    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:28.222269 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:28.222234    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:28.897851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:28.897810    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 11 12:28:28.898228 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:28.897874    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 11 12:28:29.898306 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:29.898266    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 11 12:28:29.898728 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:29.898326    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 11 12:28:30.898675 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:30.898634    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 11 12:28:30.899101 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:30.898706    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 11 12:28:31.298330 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.298226    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:31.399003 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.398966    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:31.658358 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.658311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:31.821107 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.821074    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:31.898235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.898202    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 11 12:28:31.898426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.898256    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 11 12:28:31.918442 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:31.918336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:32.016477 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.016443    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"]
Sep 11 12:28:32.020143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.020121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.023000 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.022975    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-edffad-dockercfg-fhbt4\""
Sep 11 12:28:32.023126 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.022982    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-edffad\""
Sep 11 12:28:32.032807 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.032785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"]
Sep 11 12:28:32.102515 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.102480    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"]
Sep 11 12:28:32.105217 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.105200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.118009 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.117989    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"]
Sep 11 12:28:32.185393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.185296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrzq\" (UniqueName: \"kubernetes.io/projected/9368df61-f662-4d3d-9ae4-0adaa5334eff-kube-api-access-dgrzq\") pod \"payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.185393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.185376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368df61-f662-4d3d-9ae4-0adaa5334eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.286160 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.286120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368df61-f662-4d3d-9ae4-0adaa5334eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.286351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.286192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsxp\" (UniqueName: \"kubernetes.io/projected/230ec79b-160b-4621-ae0c-17e368f30bb4-kube-api-access-wmsxp\") pod \"payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.286351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.286293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrzq\" (UniqueName: \"kubernetes.io/projected/9368df61-f662-4d3d-9ae4-0adaa5334eff-kube-api-access-dgrzq\") pod \"payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.286351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.286321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230ec79b-160b-4621-ae0c-17e368f30bb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.286928 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.286909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368df61-f662-4d3d-9ae4-0adaa5334eff-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.293821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.293799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrzq\" (UniqueName: \"kubernetes.io/projected/9368df61-f662-4d3d-9ae4-0adaa5334eff-kube-api-access-dgrzq\") pod \"payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.330876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.330849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.349039 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.348946    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:32.387566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.387492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsxp\" (UniqueName: \"kubernetes.io/projected/230ec79b-160b-4621-ae0c-17e368f30bb4-kube-api-access-wmsxp\") pod \"payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.387726 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.387580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230ec79b-160b-4621-ae0c-17e368f30bb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.388280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.388260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230ec79b-160b-4621-ae0c-17e368f30bb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.397063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.397038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsxp\" (UniqueName: \"kubernetes.io/projected/230ec79b-160b-4621-ae0c-17e368f30bb4-kube-api-access-wmsxp\") pod \"payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.416466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.416182    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.634548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.634516    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"]
Sep 11 12:28:32.634948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.634928    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:32.635973 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:32.635948    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9368df61_f662_4d3d_9ae4_0adaa5334eff.slice/crio-76c1f6206f82cd945bd4df85e8348e21af8c02adbcf10c73e3c951585f6171a8 WatchSource:0}: Error finding container 76c1f6206f82cd945bd4df85e8348e21af8c02adbcf10c73e3c951585f6171a8: Status 404 returned error can't find the container with id 76c1f6206f82cd945bd4df85e8348e21af8c02adbcf10c73e3c951585f6171a8
Sep 11 12:28:32.728461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.728435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"]
Sep 11 12:28:32.730020 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:28:32.729988    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230ec79b_160b_4621_ae0c_17e368f30bb4.slice/crio-e8c8a8d5964cb5bd28818c9e297b3e79ab88ad68282f350be9f4b9593e0785c2 WatchSource:0}: Error finding container e8c8a8d5964cb5bd28818c9e297b3e79ab88ad68282f350be9f4b9593e0785c2: Status 404 returned error can't find the container with id e8c8a8d5964cb5bd28818c9e297b3e79ab88ad68282f350be9f4b9593e0785c2
Sep 11 12:28:32.850295 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.850248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v" event={"ID":"230ec79b-160b-4621-ae0c-17e368f30bb4","Type":"ContainerStarted","Data":"9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818"}
Sep 11 12:28:32.850501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.850323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v" event={"ID":"230ec79b-160b-4621-ae0c-17e368f30bb4","Type":"ContainerStarted","Data":"e8c8a8d5964cb5bd28818c9e297b3e79ab88ad68282f350be9f4b9593e0785c2"}
Sep 11 12:28:32.851190 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.851161    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:32.853373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.853342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7" event={"ID":"9368df61-f662-4d3d-9ae4-0adaa5334eff","Type":"ContainerStarted","Data":"ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449"}
Sep 11 12:28:32.853501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.853413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7" event={"ID":"9368df61-f662-4d3d-9ae4-0adaa5334eff","Type":"ContainerStarted","Data":"76c1f6206f82cd945bd4df85e8348e21af8c02adbcf10c73e3c951585f6171a8"}
Sep 11 12:28:32.853948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.853925    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:32.870415 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.870330    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v" podStartSLOduration=0.870312774 podStartE2EDuration="870.312774ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:32.867185247 +0000 UTC m=+848.257722595" watchObservedRunningTime="2026-09-11 12:28:32.870312774 +0000 UTC m=+848.260850120"
Sep 11 12:28:32.884674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.884619    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7" podStartSLOduration=1.88460099 podStartE2EDuration="1.88460099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:32.882290795 +0000 UTC m=+848.272828129" watchObservedRunningTime="2026-09-11 12:28:32.88460099 +0000 UTC m=+848.275138333"
Sep 11 12:28:32.897855 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.897813    2561 patch_prober.go:28] interesting pod/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 11 12:28:32.898007 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:32.897921    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 11 12:28:33.096482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:33.096451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:33.223576 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:33.223538    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:33.689353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:33.689318    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:33.747248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:33.747205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:33.901896 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:33.901862    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:33.903560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:33.903522    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:28:34.090825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:34.090777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:34.720707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:34.720662    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:42.610226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:42.610160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:44.868991 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:44.868958    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:28:44.869357 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:44.869032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:28:50.178872 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.178338    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"]
Sep 11 12:28:50.178872 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.178586    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" podUID="feecc3fd-6a6b-42e3-813f-133aab831923" containerName="maas-api" containerID="cri-o://196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f" gracePeriod=30
Sep 11 12:28:50.252360 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.252324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:50.334515 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.333151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:50.400553 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.400500    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:50.450274 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.450246    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:50.459754 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.459724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:50.517839 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.517806    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:50.563982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.563936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feecc3fd-6a6b-42e3-813f-133aab831923-openshift-service-ca\") pod \"feecc3fd-6a6b-42e3-813f-133aab831923\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") "
Sep 11 12:28:50.564346 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.564310    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feecc3fd-6a6b-42e3-813f-133aab831923-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "feecc3fd-6a6b-42e3-813f-133aab831923" (UID: "feecc3fd-6a6b-42e3-813f-133aab831923"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:28:50.564559 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.564526    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls\") pod \"feecc3fd-6a6b-42e3-813f-133aab831923\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") "
Sep 11 12:28:50.564714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.564597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz6dw\" (UniqueName: \"kubernetes.io/projected/feecc3fd-6a6b-42e3-813f-133aab831923-kube-api-access-rz6dw\") pod \"feecc3fd-6a6b-42e3-813f-133aab831923\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") "
Sep 11 12:28:50.564714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.564626    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-metrics-certs\") pod \"feecc3fd-6a6b-42e3-813f-133aab831923\" (UID: \"feecc3fd-6a6b-42e3-813f-133aab831923\") "
Sep 11 12:28:50.565022 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.564995    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feecc3fd-6a6b-42e3-813f-133aab831923-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.568034 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.568005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feecc3fd-6a6b-42e3-813f-133aab831923-kube-api-access-rz6dw" (OuterVolumeSpecName: "kube-api-access-rz6dw") pod "feecc3fd-6a6b-42e3-813f-133aab831923" (UID: "feecc3fd-6a6b-42e3-813f-133aab831923"). InnerVolumeSpecName "kube-api-access-rz6dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:28:50.569741 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.569574    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "feecc3fd-6a6b-42e3-813f-133aab831923" (UID: "feecc3fd-6a6b-42e3-813f-133aab831923"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:28:50.570490 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.570467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "feecc3fd-6a6b-42e3-813f-133aab831923" (UID: "feecc3fd-6a6b-42e3-813f-133aab831923"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:28:50.602718 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.602676    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:50.666422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.666364    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.666422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.666394    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6dw\" (UniqueName: \"kubernetes.io/projected/feecc3fd-6a6b-42e3-813f-133aab831923-kube-api-access-rz6dw\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.666606 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.666439    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feecc3fd-6a6b-42e3-813f-133aab831923-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.948600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.948565    2561 generic.go:356] "Generic (PLEG): container finished" podID="feecc3fd-6a6b-42e3-813f-133aab831923" containerID="196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f" exitCode=0
Sep 11 12:28:50.948766 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.948610    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"
Sep 11 12:28:50.948766 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.948618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" event={"ID":"feecc3fd-6a6b-42e3-813f-133aab831923","Type":"ContainerDied","Data":"196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f"}
Sep 11 12:28:50.948766 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.948661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9" event={"ID":"feecc3fd-6a6b-42e3-813f-133aab831923","Type":"ContainerDied","Data":"6ec74a6c75c7a51d0c3fda854ba8063eec49a7cbf86844e1ca3d098486ac3adf"}
Sep 11 12:28:50.948766 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.948681    2561 scope.go:117] "RemoveContainer" containerID="196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f"
Sep 11 12:28:50.958319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.958304    2561 scope.go:117] "RemoveContainer" containerID="196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f"
Sep 11 12:28:50.958638 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:28:50.958620    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f\": container with ID starting with 196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f not found: ID does not exist" containerID="196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f"
Sep 11 12:28:50.958711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.958647    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f"} err="failed to get container status \"196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f\": rpc error: code = NotFound desc = could not find container \"196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f\": container with ID starting with 196ade9cb9ddaf35a9a6c8f9ef3ae7ea2d479d3e8004beacf012d72275dff65f not found: ID does not exist"
Sep 11 12:28:50.984284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.984255    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"]
Sep 11 12:28:50.990251 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:50.990228    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7e8d45b5-77ff96f486-7bgt9"]
Sep 11 12:28:51.090261 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:51.090224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:51.107050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:51.107022    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feecc3fd-6a6b-42e3-813f-133aab831923" path="/var/lib/kubelet/pods/feecc3fd-6a6b-42e3-813f-133aab831923/volumes"
Sep 11 12:28:51.190053 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:51.190020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:51.290185 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:51.290104    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:51.604701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:51.604667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:51.952654 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:51.952543    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:52.424555 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:52.424499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:54.039363 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:54.039330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:54.139863 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:54.139825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:54.393552 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:54.393516    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:54.495433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:54.495381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:54.795578 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:54.795489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:54.977684 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:54.977652    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:28:55.072286 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:28:55.072192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:12.110867 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:12.110833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:25.646716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:25.646677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:25.752197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:25.752161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:25.851017 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:25.850983    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:25.918717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:25.918623    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:26.033900 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:26.033867    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:26.460607 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:26.460572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:26.757318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:26.757232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:26.859008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:26.858969    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:29.132989 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:29.132917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:29.232416 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:29.232376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:30.277295 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.277253    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:30.330677 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.330648    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:30.541277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.541187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:30.839150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.839107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:30.939388 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.939346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:30.989055 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.989022    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"]
Sep 11 12:29:30.989488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.989475    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feecc3fd-6a6b-42e3-813f-133aab831923" containerName="maas-api"
Sep 11 12:29:30.989540 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.989489    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="feecc3fd-6a6b-42e3-813f-133aab831923" containerName="maas-api"
Sep 11 12:29:30.989575 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.989570    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="feecc3fd-6a6b-42e3-813f-133aab831923" containerName="maas-api"
Sep 11 12:29:30.992322 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.992307    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:30.994647 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.994626    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-74075c51\""
Sep 11 12:29:30.994770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:30.994676    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-74075c51-dockercfg-b8kbz\""
Sep 11 12:29:31.004086 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.004064    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"]
Sep 11 12:29:31.050513 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.050487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-74075c51-687859c947-mg484\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") " pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.050662 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.050520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gvc\" (UniqueName: \"kubernetes.io/projected/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-kube-api-access-v4gvc\") pod \"payload-processing-e2e-ait-74075c51-687859c947-mg484\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") " pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.152042 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.151954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-74075c51-687859c947-mg484\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") " pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.152309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.152283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gvc\" (UniqueName: \"kubernetes.io/projected/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-kube-api-access-v4gvc\") pod \"payload-processing-e2e-ait-74075c51-687859c947-mg484\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") " pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.152913 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.152893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-plugins-config-volume\") pod \"payload-processing-e2e-ait-74075c51-687859c947-mg484\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") " pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.161019 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.160997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gvc\" (UniqueName: \"kubernetes.io/projected/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-kube-api-access-v4gvc\") pod \"payload-processing-e2e-ait-74075c51-687859c947-mg484\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") " pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.303214 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.303179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:31.579250 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:31.579221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"]
Sep 11 12:29:31.581186 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:29:31.581154    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a746a4_652b_4a5d_a3fd_d2bb4ed647ff.slice/crio-4471e11e44c3a09c0aee28a99c9a0f47d87eae8caed5e139c3fc86810105f1c2 WatchSource:0}: Error finding container 4471e11e44c3a09c0aee28a99c9a0f47d87eae8caed5e139c3fc86810105f1c2: Status 404 returned error can't find the container with id 4471e11e44c3a09c0aee28a99c9a0f47d87eae8caed5e139c3fc86810105f1c2
Sep 11 12:29:32.164570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:32.164526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484" event={"ID":"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff","Type":"ContainerStarted","Data":"750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067"}
Sep 11 12:29:32.164751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:32.164593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484" event={"ID":"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff","Type":"ContainerStarted","Data":"4471e11e44c3a09c0aee28a99c9a0f47d87eae8caed5e139c3fc86810105f1c2"}
Sep 11 12:29:32.164751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:32.164657    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:32.183876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:32.183831    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484" podStartSLOduration=2.1838167 podStartE2EDuration="2.1838167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:32.18202546 +0000 UTC m=+907.572562805" watchObservedRunningTime="2026-09-11 12:29:32.1838167 +0000 UTC m=+907.574354042"
Sep 11 12:29:33.289697 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:33.289666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:29:43.171447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:43.171391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:29:51.658537 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:29:51.658477    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:00.165097 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.165065    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"]
Sep 11 12:30:00.168351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.168328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"
Sep 11 12:30:00.170628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.170608    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:30:00.180247 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.180220    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"
Sep 11 12:30:00.186084 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.186060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"]
Sep 11 12:30:00.496206 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.496181    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"]
Sep 11 12:30:00.497802 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:30:00.497766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f3c3c6_7871_4057_ad56_a2604731e258.slice/crio-cf5275ab3282546b89c998280f14d260f2273d5711985eafb3a77d5d28bb9520 WatchSource:0}: Error finding container cf5275ab3282546b89c998280f14d260f2273d5711985eafb3a77d5d28bb9520: Status 404 returned error can't find the container with id cf5275ab3282546b89c998280f14d260f2273d5711985eafb3a77d5d28bb9520
Sep 11 12:30:00.799345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.799270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:00.824425 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.824377    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"]
Sep 11 12:30:00.824658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.824630    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484" podUID="97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" containerName="payload-processing" containerID="cri-o://750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067" gracePeriod=30
Sep 11 12:30:00.899824 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.899789    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:00.997803 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:00.997767    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:01.063954 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.063917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:01.070785 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.070762    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:30:01.138699 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.138668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-plugins-config-volume\") pod \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") "
Sep 11 12:30:01.138875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.138775    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4gvc\" (UniqueName: \"kubernetes.io/projected/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-kube-api-access-v4gvc\") pod \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\" (UID: \"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff\") "
Sep 11 12:30:01.139094 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.139069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" (UID: "97a746a4-652b-4a5d-a3fd-d2bb4ed647ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:01.140945 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.140918    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-kube-api-access-v4gvc" (OuterVolumeSpecName: "kube-api-access-v4gvc") pod "97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" (UID: "97a746a4-652b-4a5d-a3fd-d2bb4ed647ff"). InnerVolumeSpecName "kube-api-access-v4gvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:01.240386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.240351    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.240386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.240387    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gvc\" (UniqueName: \"kubernetes.io/projected/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff-kube-api-access-v4gvc\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.311248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.311165    2561 generic.go:356] "Generic (PLEG): container finished" podID="97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" containerID="750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067" exitCode=0
Sep 11 12:30:01.311248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.311205    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"
Sep 11 12:30:01.311248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.311216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484" event={"ID":"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff","Type":"ContainerDied","Data":"750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067"}
Sep 11 12:30:01.311548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.311280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484" event={"ID":"97a746a4-652b-4a5d-a3fd-d2bb4ed647ff","Type":"ContainerDied","Data":"4471e11e44c3a09c0aee28a99c9a0f47d87eae8caed5e139c3fc86810105f1c2"}
Sep 11 12:30:01.311548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.311310    2561 scope.go:117] "RemoveContainer" containerID="750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067"
Sep 11 12:30:01.312708 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.312644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d" event={"ID":"59f3c3c6-7871-4057-ad56-a2604731e258","Type":"ContainerStarted","Data":"cf5275ab3282546b89c998280f14d260f2273d5711985eafb3a77d5d28bb9520"}
Sep 11 12:30:01.321363 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.321343    2561 scope.go:117] "RemoveContainer" containerID="750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067"
Sep 11 12:30:01.321792 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:30:01.321764    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067\": container with ID starting with 750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067 not found: ID does not exist" containerID="750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067"
Sep 11 12:30:01.321881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.321803    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067"} err="failed to get container status \"750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067\": rpc error: code = NotFound desc = could not find container \"750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067\": container with ID starting with 750ddc7a26681a08171866da0579a137c9a5d79e24c7b93978d1e5a531b7f067 not found: ID does not exist"
Sep 11 12:30:01.340346 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.340326    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"]
Sep 11 12:30:01.348542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:01.348522    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-74075c51-687859c947-mg484"]
Sep 11 12:30:02.104802 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:02.104768    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:02.521284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:02.521183    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:02.995878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:02.995796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:03.108983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:03.108953    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" path="/var/lib/kubelet/pods/97a746a4-652b-4a5d-a3fd-d2bb4ed647ff/volumes"
Sep 11 12:30:03.167967 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:03.167931    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:04.339179 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:04.339143    2561 generic.go:356] "Generic (PLEG): container finished" podID="59f3c3c6-7871-4057-ad56-a2604731e258" containerID="a89cfe19bb58d2e4952c164388f98eaf33e7f92dc26ab7c0d92d06f6034aee75" exitCode=0
Sep 11 12:30:04.339560 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:04.339231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d" event={"ID":"59f3c3c6-7871-4057-ad56-a2604731e258","Type":"ContainerDied","Data":"a89cfe19bb58d2e4952c164388f98eaf33e7f92dc26ab7c0d92d06f6034aee75"}
Sep 11 12:30:05.385589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:05.385551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:05.480085 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:05.480063    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"
Sep 11 12:30:06.087090 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:06.087059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:06.189446 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:06.189418    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:06.353215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:06.353139    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"
Sep 11 12:30:06.353215 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:06.353178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d" event={"ID":"59f3c3c6-7871-4057-ad56-a2604731e258","Type":"ContainerDied","Data":"cf5275ab3282546b89c998280f14d260f2273d5711985eafb3a77d5d28bb9520"}
Sep 11 12:30:06.353436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:06.353249    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf5275ab3282546b89c998280f14d260f2273d5711985eafb3a77d5d28bb9520"
Sep 11 12:30:06.426209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:06.426176    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:07.336480 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.336450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:07.397957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.397924    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:07.586281 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.586249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:07.791511 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.791410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:07.853891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.853856    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:07.934199 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934167    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"]
Sep 11 12:30:07.934607 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934593    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f3c3c6-7871-4057-ad56-a2604731e258" containerName="cleanup"
Sep 11 12:30:07.934677 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934609    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f3c3c6-7871-4057-ad56-a2604731e258" containerName="cleanup"
Sep 11 12:30:07.934677 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934636    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" containerName="payload-processing"
Sep 11 12:30:07.934677 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934642    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" containerName="payload-processing"
Sep 11 12:30:07.934768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934704    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f3c3c6-7871-4057-ad56-a2604731e258" containerName="cleanup"
Sep 11 12:30:07.934768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.934715    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a746a4-652b-4a5d-a3fd-d2bb4ed647ff" containerName="payload-processing"
Sep 11 12:30:07.936806 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.936789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:07.939214 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.939186    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-411110ed-dockercfg-fnxjh\""
Sep 11 12:30:07.939214 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.939190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-411110ed\""
Sep 11 12:30:07.947720 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:07.947697    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"]
Sep 11 12:30:08.007275 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.007240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-plugins-config-volume\") pod \"payload-processing-e2e-derive-411110ed-f984c994f-zvwf7\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") " pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.007461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.007286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlxp\" (UniqueName: \"kubernetes.io/projected/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-kube-api-access-pzlxp\") pod \"payload-processing-e2e-derive-411110ed-f984c994f-zvwf7\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") " pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.108724 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.108681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-plugins-config-volume\") pod \"payload-processing-e2e-derive-411110ed-f984c994f-zvwf7\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") " pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.108906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.108741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlxp\" (UniqueName: \"kubernetes.io/projected/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-kube-api-access-pzlxp\") pod \"payload-processing-e2e-derive-411110ed-f984c994f-zvwf7\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") " pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.109717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.109693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-plugins-config-volume\") pod \"payload-processing-e2e-derive-411110ed-f984c994f-zvwf7\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") " pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.119372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.119342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlxp\" (UniqueName: \"kubernetes.io/projected/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-kube-api-access-pzlxp\") pod \"payload-processing-e2e-derive-411110ed-f984c994f-zvwf7\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") " pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.216035 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.215994    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:08.249536 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.249496    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:08.523205 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:08.523182    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"]
Sep 11 12:30:08.525547 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:30:08.525519    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11a53fb_04bf_4b1b_b870_e54251ea5d8d.slice/crio-eae324e6bd44e62a25e437de26b1fb27b511b494913eb66af6720c2c2e5a69ef WatchSource:0}: Error finding container eae324e6bd44e62a25e437de26b1fb27b511b494913eb66af6720c2c2e5a69ef: Status 404 returned error can't find the container with id eae324e6bd44e62a25e437de26b1fb27b511b494913eb66af6720c2c2e5a69ef
Sep 11 12:30:09.037311 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:09.037276    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:09.371687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:09.371645    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7" event={"ID":"a11a53fb-04bf-4b1b-b870-e54251ea5d8d","Type":"ContainerStarted","Data":"cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57"}
Sep 11 12:30:09.371881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:09.371715    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:09.371881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:09.371740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7" event={"ID":"a11a53fb-04bf-4b1b-b870-e54251ea5d8d","Type":"ContainerStarted","Data":"eae324e6bd44e62a25e437de26b1fb27b511b494913eb66af6720c2c2e5a69ef"}
Sep 11 12:30:09.390690 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:09.390629    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7" podStartSLOduration=2.390610518 podStartE2EDuration="2.390610518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:09.387122701 +0000 UTC m=+944.777660043" watchObservedRunningTime="2026-09-11 12:30:09.390610518 +0000 UTC m=+944.781147860"
Sep 11 12:30:16.453585 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:16.453550    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:17.222805 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.222769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:17.277052 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.277021    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"]
Sep 11 12:30:17.277332 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.277299    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95" podUID="a6148102-bff0-4869-baf8-2482e8d6714e" containerName="payload-processing" containerID="cri-o://3b81ae87f4c7fdb95b758228a735bd2ef98127030ab5a7e3fe4e3c46de3be5f9" gracePeriod=30
Sep 11 12:30:17.297462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.297255    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"]
Sep 11 12:30:17.299428 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.297664    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv" podUID="b5114de1-a9fb-4d83-b5ac-c4a7646d950d" containerName="payload-pre-processing" containerID="cri-o://87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9" gracePeriod=30
Sep 11 12:30:17.318390 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.318357    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:17.415319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.415288    2561 generic.go:356] "Generic (PLEG): container finished" podID="a6148102-bff0-4869-baf8-2482e8d6714e" containerID="3b81ae87f4c7fdb95b758228a735bd2ef98127030ab5a7e3fe4e3c46de3be5f9" exitCode=0
Sep 11 12:30:17.415468 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.415337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95" event={"ID":"a6148102-bff0-4869-baf8-2482e8d6714e","Type":"ContainerDied","Data":"3b81ae87f4c7fdb95b758228a735bd2ef98127030ab5a7e3fe4e3c46de3be5f9"}
Sep 11 12:30:17.417143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.417114    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:17.519594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.519521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:17.596948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.596920    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:30:17.600379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.600358    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:30:17.705850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.705819    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsnmk\" (UniqueName: \"kubernetes.io/projected/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-kube-api-access-lsnmk\") pod \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") "
Sep 11 12:30:17.706024 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.705880    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-plugins-config-volume\") pod \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\" (UID: \"b5114de1-a9fb-4d83-b5ac-c4a7646d950d\") "
Sep 11 12:30:17.706024 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.705942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6148102-bff0-4869-baf8-2482e8d6714e-plugins-config-volume\") pod \"a6148102-bff0-4869-baf8-2482e8d6714e\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") "
Sep 11 12:30:17.706153 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.706066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw275\" (UniqueName: \"kubernetes.io/projected/a6148102-bff0-4869-baf8-2482e8d6714e-kube-api-access-gw275\") pod \"a6148102-bff0-4869-baf8-2482e8d6714e\" (UID: \"a6148102-bff0-4869-baf8-2482e8d6714e\") "
Sep 11 12:30:17.706253 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.706227    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5114de1-a9fb-4d83-b5ac-c4a7646d950d" (UID: "b5114de1-a9fb-4d83-b5ac-c4a7646d950d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:17.706327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.706307    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6148102-bff0-4869-baf8-2482e8d6714e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6148102-bff0-4869-baf8-2482e8d6714e" (UID: "a6148102-bff0-4869-baf8-2482e8d6714e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:17.706461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.706438    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.706461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.706462    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6148102-bff0-4869-baf8-2482e8d6714e-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.708041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.708017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-kube-api-access-lsnmk" (OuterVolumeSpecName: "kube-api-access-lsnmk") pod "b5114de1-a9fb-4d83-b5ac-c4a7646d950d" (UID: "b5114de1-a9fb-4d83-b5ac-c4a7646d950d"). InnerVolumeSpecName "kube-api-access-lsnmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:17.708041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.708035    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6148102-bff0-4869-baf8-2482e8d6714e-kube-api-access-gw275" (OuterVolumeSpecName: "kube-api-access-gw275") pod "a6148102-bff0-4869-baf8-2482e8d6714e" (UID: "a6148102-bff0-4869-baf8-2482e8d6714e"). InnerVolumeSpecName "kube-api-access-gw275". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:17.807984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.807901    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw275\" (UniqueName: \"kubernetes.io/projected/a6148102-bff0-4869-baf8-2482e8d6714e-kube-api-access-gw275\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.807984 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.807931    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnmk\" (UniqueName: \"kubernetes.io/projected/b5114de1-a9fb-4d83-b5ac-c4a7646d950d-kube-api-access-lsnmk\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.859280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:17.859247    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:18.421667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.421639    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"
Sep 11 12:30:18.421851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.421641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95" event={"ID":"a6148102-bff0-4869-baf8-2482e8d6714e","Type":"ContainerDied","Data":"2ac3d1bf2834f7af0ca786810f7081b90483d5aacdeb3f015f035eb4fd7533f5"}
Sep 11 12:30:18.421851 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.421771    2561 scope.go:117] "RemoveContainer" containerID="3b81ae87f4c7fdb95b758228a735bd2ef98127030ab5a7e3fe4e3c46de3be5f9"
Sep 11 12:30:18.423121 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.423022    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5114de1-a9fb-4d83-b5ac-c4a7646d950d" containerID="87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9" exitCode=0
Sep 11 12:30:18.423121 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.423075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv" event={"ID":"b5114de1-a9fb-4d83-b5ac-c4a7646d950d","Type":"ContainerDied","Data":"87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9"}
Sep 11 12:30:18.423121 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.423081    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"
Sep 11 12:30:18.423121 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.423115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv" event={"ID":"b5114de1-a9fb-4d83-b5ac-c4a7646d950d","Type":"ContainerDied","Data":"417633a3925ad2ae205cb81fac31e3513813b561ed393206bb8b4774f9e31bba"}
Sep 11 12:30:18.432531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.432376    2561 scope.go:117] "RemoveContainer" containerID="87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9"
Sep 11 12:30:18.441380 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.441358    2561 scope.go:117] "RemoveContainer" containerID="87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9"
Sep 11 12:30:18.441728 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:30:18.441700    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9\": container with ID starting with 87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9 not found: ID does not exist" containerID="87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9"
Sep 11 12:30:18.441808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.441734    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9"} err="failed to get container status \"87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9\": rpc error: code = NotFound desc = could not find container \"87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9\": container with ID starting with 87d229a8bbda272c29a6835ffd84796749e383e2dc29d58b12f6b5352fe2bfd9 not found: ID does not exist"
Sep 11 12:30:18.447164 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.447143    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"]
Sep 11 12:30:18.453477 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.453455    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9226c5-77f966c67-zhb95"]
Sep 11 12:30:18.468092 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.468071    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"]
Sep 11 12:30:18.472894 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:18.472874    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9226c5-7c8f78478b-xxgxv"]
Sep 11 12:30:19.106653 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:19.106614    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6148102-bff0-4869-baf8-2482e8d6714e" path="/var/lib/kubelet/pods/a6148102-bff0-4869-baf8-2482e8d6714e/volumes"
Sep 11 12:30:19.107050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:19.107035    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5114de1-a9fb-4d83-b5ac-c4a7646d950d" path="/var/lib/kubelet/pods/b5114de1-a9fb-4d83-b5ac-c4a7646d950d/volumes"
Sep 11 12:30:20.379300 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:20.379272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:22.887585 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:22.887539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:23.198987 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:23.198882    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:26.898980 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:26.898942    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:28.330676 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:28.330641    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:33.543393 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:33.543355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:37.226828 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.226795    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:37.269563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.269529    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"]
Sep 11 12:30:37.269882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.269850    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7" podUID="a11a53fb-04bf-4b1b-b870-e54251ea5d8d" containerName="payload-processing" containerID="cri-o://cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57" gracePeriod=30
Sep 11 12:30:37.325985 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.325951    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:37.426834 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.426797    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:37.506206 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.506179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:37.522794 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.522771    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:37.527548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.527521    2561 generic.go:356] "Generic (PLEG): container finished" podID="a11a53fb-04bf-4b1b-b870-e54251ea5d8d" containerID="cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57" exitCode=0
Sep 11 12:30:37.527687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.527558    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"
Sep 11 12:30:37.527687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.527602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7" event={"ID":"a11a53fb-04bf-4b1b-b870-e54251ea5d8d","Type":"ContainerDied","Data":"cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57"}
Sep 11 12:30:37.527687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.527657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7" event={"ID":"a11a53fb-04bf-4b1b-b870-e54251ea5d8d","Type":"ContainerDied","Data":"eae324e6bd44e62a25e437de26b1fb27b511b494913eb66af6720c2c2e5a69ef"}
Sep 11 12:30:37.527822 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.527692    2561 scope.go:117] "RemoveContainer" containerID="cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57"
Sep 11 12:30:37.538113 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.538089    2561 scope.go:117] "RemoveContainer" containerID="cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57"
Sep 11 12:30:37.538469 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:30:37.538448    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57\": container with ID starting with cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57 not found: ID does not exist" containerID="cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57"
Sep 11 12:30:37.538558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.538481    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57"} err="failed to get container status \"cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57\": rpc error: code = NotFound desc = could not find container \"cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57\": container with ID starting with cf51578d1be9b9b79f5532b41cc37edf43927e5d84c5cfd0a2d802540afffb57 not found: ID does not exist"
Sep 11 12:30:37.597332 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.597303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-plugins-config-volume\") pod \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") "
Sep 11 12:30:37.597544 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.597346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzlxp\" (UniqueName: \"kubernetes.io/projected/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-kube-api-access-pzlxp\") pod \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\" (UID: \"a11a53fb-04bf-4b1b-b870-e54251ea5d8d\") "
Sep 11 12:30:37.597716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.597691    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a11a53fb-04bf-4b1b-b870-e54251ea5d8d" (UID: "a11a53fb-04bf-4b1b-b870-e54251ea5d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:37.599492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.599467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-kube-api-access-pzlxp" (OuterVolumeSpecName: "kube-api-access-pzlxp") pod "a11a53fb-04bf-4b1b-b870-e54251ea5d8d" (UID: "a11a53fb-04bf-4b1b-b870-e54251ea5d8d"). InnerVolumeSpecName "kube-api-access-pzlxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:37.662548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.662510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:37.698925 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.698898    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.698925 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.698923    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzlxp\" (UniqueName: \"kubernetes.io/projected/a11a53fb-04bf-4b1b-b870-e54251ea5d8d-kube-api-access-pzlxp\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.852725 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.852692    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"]
Sep 11 12:30:37.858481 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:37.858456    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-411110ed-f984c994f-zvwf7"]
Sep 11 12:30:38.398732 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:38.398695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:38.466312 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:38.466275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:38.535026 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:38.534988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:39.113445 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:39.113411    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11a53fb-04bf-4b1b-b870-e54251ea5d8d" path="/var/lib/kubelet/pods/a11a53fb-04bf-4b1b-b870-e54251ea5d8d/volumes"
Sep 11 12:30:40.163345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:40.163308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:46.486322 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:46.486283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:46.589826 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:46.589778    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:46.787720 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:46.787616    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:46.921689 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:46.921657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:30:47.428191 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:30:47.428113    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:07.542686 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:07.542649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:07.828822 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:07.828788    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:08.341930 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:08.341888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:08.794152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:08.794063    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:08.911441 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:08.911389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:11.549609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:11.549577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:11.641289 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:11.641251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:12.681747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.681703    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"]
Sep 11 12:31:12.682336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682317    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6148102-bff0-4869-baf8-2482e8d6714e" containerName="payload-processing"
Sep 11 12:31:12.682336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682336    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6148102-bff0-4869-baf8-2482e8d6714e" containerName="payload-processing"
Sep 11 12:31:12.682509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682359    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11a53fb-04bf-4b1b-b870-e54251ea5d8d" containerName="payload-processing"
Sep 11 12:31:12.682509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682367    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11a53fb-04bf-4b1b-b870-e54251ea5d8d" containerName="payload-processing"
Sep 11 12:31:12.682509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682411    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5114de1-a9fb-4d83-b5ac-c4a7646d950d" containerName="payload-pre-processing"
Sep 11 12:31:12.682509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682420    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5114de1-a9fb-4d83-b5ac-c4a7646d950d" containerName="payload-pre-processing"
Sep 11 12:31:12.682509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682507    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11a53fb-04bf-4b1b-b870-e54251ea5d8d" containerName="payload-processing"
Sep 11 12:31:12.682796 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682519    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5114de1-a9fb-4d83-b5ac-c4a7646d950d" containerName="payload-pre-processing"
Sep 11 12:31:12.682796 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.682537    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6148102-bff0-4869-baf8-2482e8d6714e" containerName="payload-processing"
Sep 11 12:31:12.686308 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.686281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.696333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.689591    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-82ead5-serving-cert\""
Sep 11 12:31:12.696333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.695103    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"]
Sep 11 12:31:12.709844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.709763    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:12.747527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.747496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2738a91-74d8-40e4-bcff-de060f2100f7-openshift-service-ca\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.747655 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.747563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dd9\" (UniqueName: \"kubernetes.io/projected/b2738a91-74d8-40e4-bcff-de060f2100f7-kube-api-access-74dd9\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.747655 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.747605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-metrics-certs\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.747734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.747655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.806566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.806523    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:12.849171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.849138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dd9\" (UniqueName: \"kubernetes.io/projected/b2738a91-74d8-40e4-bcff-de060f2100f7-kube-api-access-74dd9\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.849309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.849182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-metrics-certs\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.849309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.849230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.849309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.849275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2738a91-74d8-40e4-bcff-de060f2100f7-openshift-service-ca\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.849533 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:31:12.849433    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-serving-cert: secret "maas-api-e2e-explicit-82ead5-serving-cert" not found
Sep 11 12:31:12.849533 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:31:12.849523    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls podName:b2738a91-74d8-40e4-bcff-de060f2100f7 nodeName:}" failed. No retries permitted until 2026-09-11 12:31:13.349501207 +0000 UTC m=+1008.740038542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls") pod "maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" (UID: "b2738a91-74d8-40e4-bcff-de060f2100f7") : secret "maas-api-e2e-explicit-82ead5-serving-cert" not found
Sep 11 12:31:12.849955 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.849936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2738a91-74d8-40e4-bcff-de060f2100f7-openshift-service-ca\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.851792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.851770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-metrics-certs\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:12.858025 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:12.858000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dd9\" (UniqueName: \"kubernetes.io/projected/b2738a91-74d8-40e4-bcff-de060f2100f7-kube-api-access-74dd9\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:13.074466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.074426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:13.193832 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.193794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:13.297159 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.297121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:13.309807 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.309776    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"]
Sep 11 12:31:13.313842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.313821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.316941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.316921    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-82ead5-dockercfg-6dqdn\""
Sep 11 12:31:13.317038 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.316976    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-82ead5\""
Sep 11 12:31:13.324856 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.324751    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"]
Sep 11 12:31:13.354596 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.354553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96639c-eca1-489b-9981-acdae271c5b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.354773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.354737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4699k\" (UniqueName: \"kubernetes.io/projected/3a96639c-eca1-489b-9981-acdae271c5b0-kube-api-access-4699k\") pod \"payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.354850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.354816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:13.357180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.357158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls\") pod \"maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:13.420784 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.420747    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:13.455622 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.455583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4699k\" (UniqueName: \"kubernetes.io/projected/3a96639c-eca1-489b-9981-acdae271c5b0-kube-api-access-4699k\") pod \"payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.455817 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.455663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96639c-eca1-489b-9981-acdae271c5b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.456291 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.456270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96639c-eca1-489b-9981-acdae271c5b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.464115 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.464085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4699k\" (UniqueName: \"kubernetes.io/projected/3a96639c-eca1-489b-9981-acdae271c5b0-kube-api-access-4699k\") pod \"payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.607089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.607001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:13.626356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.626324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:13.887266 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.887234    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"]
Sep 11 12:31:13.888945 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:31:13.888913    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2738a91_74d8_40e4_bcff_de060f2100f7.slice/crio-df4aab6e6ee48b225dddc167cbcc5b88e2853f75243c544d344885fab0a45bf6 WatchSource:0}: Error finding container df4aab6e6ee48b225dddc167cbcc5b88e2853f75243c544d344885fab0a45bf6: Status 404 returned error can't find the container with id df4aab6e6ee48b225dddc167cbcc5b88e2853f75243c544d344885fab0a45bf6
Sep 11 12:31:13.890646 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.890614    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:31:13.917770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:13.917743    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"]
Sep 11 12:31:13.919511 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:31:13.919486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a96639c_eca1_489b_9981_acdae271c5b0.slice/crio-aba6d650fb9e258b19172d5396e5a2b20f6217d3229e73aaf4f1cc029b41be69 WatchSource:0}: Error finding container aba6d650fb9e258b19172d5396e5a2b20f6217d3229e73aaf4f1cc029b41be69: Status 404 returned error can't find the container with id aba6d650fb9e258b19172d5396e5a2b20f6217d3229e73aaf4f1cc029b41be69
Sep 11 12:31:14.725447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.725372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j" event={"ID":"3a96639c-eca1-489b-9981-acdae271c5b0","Type":"ContainerStarted","Data":"466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521"}
Sep 11 12:31:14.725659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.725469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:14.725659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.725495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j" event={"ID":"3a96639c-eca1-489b-9981-acdae271c5b0","Type":"ContainerStarted","Data":"aba6d650fb9e258b19172d5396e5a2b20f6217d3229e73aaf4f1cc029b41be69"}
Sep 11 12:31:14.727128 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.727099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" event={"ID":"b2738a91-74d8-40e4-bcff-de060f2100f7","Type":"ContainerStarted","Data":"cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86"}
Sep 11 12:31:14.727237 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.727152    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:14.727237 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.727174    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" event={"ID":"b2738a91-74d8-40e4-bcff-de060f2100f7","Type":"ContainerStarted","Data":"df4aab6e6ee48b225dddc167cbcc5b88e2853f75243c544d344885fab0a45bf6"}
Sep 11 12:31:14.750238 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.750183    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j" podStartSLOduration=1.7501655729999999 podStartE2EDuration="1.750165573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:14.746968699 +0000 UTC m=+1010.137506065" watchObservedRunningTime="2026-09-11 12:31:14.750165573 +0000 UTC m=+1010.140702948"
Sep 11 12:31:14.768702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:14.768646    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" podStartSLOduration=2.056169763 podStartE2EDuration="2.76862819s" totalImagesPullingTime="712.458427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:14.767019305 +0000 UTC m=+1010.157556650" watchObservedRunningTime="2026-09-11 12:31:14.76862819 +0000 UTC m=+1010.159165534"
Sep 11 12:31:19.470782 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:19.470748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:19.542860 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:19.542812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:19.631346 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:19.631296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:19.724541 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:19.724395    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:20.737310 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:20.737282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:31:25.733902 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:25.733870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:31:35.637451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:35.637394    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:36.590887 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:36.590853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:39.692226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:39.692184    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:41.593910 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:41.593875    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:41.664640 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:41.664595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:41.752874 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:41.752835    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:41.850844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:41.850752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:50.535749 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:50.535713    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:52.341759 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:52.341711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:53.148536 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:53.148504    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:58.101472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:58.101440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:59.310420 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:59.310376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:59.411297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:59.411264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:59.513021 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:59.512982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:31:59.612308 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:31:59.612273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:00.305009 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:00.304956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:00.396878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:00.396841    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:00.454073 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:00.454018    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:02.007693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:02.007653    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:02.109092 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:02.109055    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:08.139773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:08.139740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:08.224703 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:08.224618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:08.829352 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:08.829306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:14.896816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:14.896775    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:14.997551 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:14.997519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:15.188793 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:15.188695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:15.446745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:15.446610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:15.603234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:15.603194    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:20.703100 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:20.703011    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:21.500440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:21.500391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:22.239764 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.239685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:22.340830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.340796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:22.861755 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.861726    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"]
Sep 11 12:32:22.865773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.865750    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:22.869155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.869100    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ad618118-serving-cert\""
Sep 11 12:32:22.875041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.875016    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"]
Sep 11 12:32:22.897540 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.897512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:22.961644 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:22.961588    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:23.009807 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.009779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/d1c14e28-4755-4043-b936-7de6ffd33ad8-kube-api-access-dqfw8\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.009967 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.009820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14e28-4755-4043-b936-7de6ffd33ad8-openshift-service-ca\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.009967 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.009949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-metrics-certs\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.010079 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.009988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.110478 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.110446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-metrics-certs\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.110660 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.110489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.110660 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.110547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/d1c14e28-4755-4043-b936-7de6ffd33ad8-kube-api-access-dqfw8\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.110660 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.110569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14e28-4755-4043-b936-7de6ffd33ad8-openshift-service-ca\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.110829 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:32:23.110671    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-serving-cert: secret "maas-api-e2e-mt-ad618118-serving-cert" not found
Sep 11 12:32:23.110829 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:32:23.110724    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls podName:d1c14e28-4755-4043-b936-7de6ffd33ad8 nodeName:}" failed. No retries permitted until 2026-09-11 12:32:23.610709385 +0000 UTC m=+1079.001246708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls") pod "maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" (UID: "d1c14e28-4755-4043-b936-7de6ffd33ad8") : secret "maas-api-e2e-mt-ad618118-serving-cert" not found
Sep 11 12:32:23.111210 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.111192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14e28-4755-4043-b936-7de6ffd33ad8-openshift-service-ca\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.113042 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.112994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-metrics-certs\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.123251 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.123223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/d1c14e28-4755-4043-b936-7de6ffd33ad8-kube-api-access-dqfw8\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.207645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.207595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:23.391610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.391521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:23.488208 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.488175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:23.615724 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.615695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.618000 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.617973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls\") pod \"maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:23.780242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:23.780163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:24.065620 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:24.065588    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"]
Sep 11 12:32:24.066931 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:32:24.066899    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c14e28_4755_4043_b936_7de6ffd33ad8.slice/crio-adad7505404c9325fe59501eaa58ef65435b46fbde488c2398d7112abda90ed9 WatchSource:0}: Error finding container adad7505404c9325fe59501eaa58ef65435b46fbde488c2398d7112abda90ed9: Status 404 returned error can't find the container with id adad7505404c9325fe59501eaa58ef65435b46fbde488c2398d7112abda90ed9
Sep 11 12:32:24.091884 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:24.091857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" event={"ID":"d1c14e28-4755-4043-b936-7de6ffd33ad8","Type":"ContainerStarted","Data":"adad7505404c9325fe59501eaa58ef65435b46fbde488c2398d7112abda90ed9"}
Sep 11 12:32:25.099409 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:25.099366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" event={"ID":"d1c14e28-4755-4043-b936-7de6ffd33ad8","Type":"ContainerStarted","Data":"b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154"}
Sep 11 12:32:25.099755 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:25.099467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:25.122531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:25.122477    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" podStartSLOduration=2.541495803 podStartE2EDuration="3.122461325s" totalImagesPullingTime="580.965522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:25.120650074 +0000 UTC m=+1080.511187417" watchObservedRunningTime="2026-09-11 12:32:25.122461325 +0000 UTC m=+1080.512998666"
Sep 11 12:32:30.944185 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:30.944149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:31.110707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.110679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:32:31.517226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.517188    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"]
Sep 11 12:32:31.517496 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.517465    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" podUID="b2738a91-74d8-40e4-bcff-de060f2100f7" containerName="maas-api" containerID="cri-o://cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86" gracePeriod=30
Sep 11 12:32:31.548783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.548748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:31.589149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.589117    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"]
Sep 11 12:32:31.589473 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.589440    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j" podUID="3a96639c-eca1-489b-9981-acdae271c5b0" containerName="payload-pre-processing" containerID="cri-o://466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521" gracePeriod=30
Sep 11 12:32:31.612064 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.612022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:31.674566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.674531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:31.747662 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.747599    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:31.802812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.802722    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:31.851919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.851892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:32:31.887357 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.887330    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:32:31.997171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74dd9\" (UniqueName: \"kubernetes.io/projected/b2738a91-74d8-40e4-bcff-de060f2100f7-kube-api-access-74dd9\") pod \"b2738a91-74d8-40e4-bcff-de060f2100f7\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") "
Sep 11 12:32:31.997682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2738a91-74d8-40e4-bcff-de060f2100f7-openshift-service-ca\") pod \"b2738a91-74d8-40e4-bcff-de060f2100f7\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") "
Sep 11 12:32:31.997682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-metrics-certs\") pod \"b2738a91-74d8-40e4-bcff-de060f2100f7\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") "
Sep 11 12:32:31.997682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls\") pod \"b2738a91-74d8-40e4-bcff-de060f2100f7\" (UID: \"b2738a91-74d8-40e4-bcff-de060f2100f7\") "
Sep 11 12:32:31.997682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96639c-eca1-489b-9981-acdae271c5b0-plugins-config-volume\") pod \"3a96639c-eca1-489b-9981-acdae271c5b0\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") "
Sep 11 12:32:31.997682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997379    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4699k\" (UniqueName: \"kubernetes.io/projected/3a96639c-eca1-489b-9981-acdae271c5b0-kube-api-access-4699k\") pod \"3a96639c-eca1-489b-9981-acdae271c5b0\" (UID: \"3a96639c-eca1-489b-9981-acdae271c5b0\") "
Sep 11 12:32:31.997961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2738a91-74d8-40e4-bcff-de060f2100f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2738a91-74d8-40e4-bcff-de060f2100f7" (UID: "b2738a91-74d8-40e4-bcff-de060f2100f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:32:31.997961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.997742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a96639c-eca1-489b-9981-acdae271c5b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a96639c-eca1-489b-9981-acdae271c5b0" (UID: "3a96639c-eca1-489b-9981-acdae271c5b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:32:31.999451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.999417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2738a91-74d8-40e4-bcff-de060f2100f7-kube-api-access-74dd9" (OuterVolumeSpecName: "kube-api-access-74dd9") pod "b2738a91-74d8-40e4-bcff-de060f2100f7" (UID: "b2738a91-74d8-40e4-bcff-de060f2100f7"). InnerVolumeSpecName "kube-api-access-74dd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:32:31.999451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.999436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2738a91-74d8-40e4-bcff-de060f2100f7" (UID: "b2738a91-74d8-40e4-bcff-de060f2100f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:32:31.999600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.999514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b2738a91-74d8-40e4-bcff-de060f2100f7" (UID: "b2738a91-74d8-40e4-bcff-de060f2100f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:32:31.999600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:31.999552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a96639c-eca1-489b-9981-acdae271c5b0-kube-api-access-4699k" (OuterVolumeSpecName: "kube-api-access-4699k") pod "3a96639c-eca1-489b-9981-acdae271c5b0" (UID: "3a96639c-eca1-489b-9981-acdae271c5b0"). InnerVolumeSpecName "kube-api-access-4699k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:32:32.097333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.097300    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:32.098514 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.098491    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a96639c-eca1-489b-9981-acdae271c5b0-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.098514 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.098514    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4699k\" (UniqueName: \"kubernetes.io/projected/3a96639c-eca1-489b-9981-acdae271c5b0-kube-api-access-4699k\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.098667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.098524    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74dd9\" (UniqueName: \"kubernetes.io/projected/b2738a91-74d8-40e4-bcff-de060f2100f7-kube-api-access-74dd9\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.098667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.098534    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2738a91-74d8-40e4-bcff-de060f2100f7-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.098667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.098543    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.098667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.098551    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2738a91-74d8-40e4-bcff-de060f2100f7-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.137486 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.137457    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a96639c-eca1-489b-9981-acdae271c5b0" containerID="466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521" exitCode=0
Sep 11 12:32:32.137642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.137514    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"
Sep 11 12:32:32.137642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.137548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j" event={"ID":"3a96639c-eca1-489b-9981-acdae271c5b0","Type":"ContainerDied","Data":"466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521"}
Sep 11 12:32:32.137642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.137606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j" event={"ID":"3a96639c-eca1-489b-9981-acdae271c5b0","Type":"ContainerDied","Data":"aba6d650fb9e258b19172d5396e5a2b20f6217d3229e73aaf4f1cc029b41be69"}
Sep 11 12:32:32.137811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.137645    2561 scope.go:117] "RemoveContainer" containerID="466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521"
Sep 11 12:32:32.139248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.139216    2561 generic.go:356] "Generic (PLEG): container finished" podID="b2738a91-74d8-40e4-bcff-de060f2100f7" containerID="cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86" exitCode=0
Sep 11 12:32:32.139376 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.139260    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"
Sep 11 12:32:32.139376 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.139311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" event={"ID":"b2738a91-74d8-40e4-bcff-de060f2100f7","Type":"ContainerDied","Data":"cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86"}
Sep 11 12:32:32.139529 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.139384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl" event={"ID":"b2738a91-74d8-40e4-bcff-de060f2100f7","Type":"ContainerDied","Data":"df4aab6e6ee48b225dddc167cbcc5b88e2853f75243c544d344885fab0a45bf6"}
Sep 11 12:32:32.148158 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.148142    2561 scope.go:117] "RemoveContainer" containerID="466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521"
Sep 11 12:32:32.148506 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:32:32.148488    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521\": container with ID starting with 466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521 not found: ID does not exist" containerID="466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521"
Sep 11 12:32:32.148598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.148516    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521"} err="failed to get container status \"466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521\": rpc error: code = NotFound desc = could not find container \"466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521\": container with ID starting with 466d9079515a6c8eceb0d7776dc7557a077dd2677dada4ddf77957167eba8521 not found: ID does not exist"
Sep 11 12:32:32.148598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.148537    2561 scope.go:117] "RemoveContainer" containerID="cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86"
Sep 11 12:32:32.157036 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.157021    2561 scope.go:117] "RemoveContainer" containerID="cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86"
Sep 11 12:32:32.157272 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:32:32.157257    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86\": container with ID starting with cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86 not found: ID does not exist" containerID="cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86"
Sep 11 12:32:32.157308 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.157278    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86"} err="failed to get container status \"cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86\": rpc error: code = NotFound desc = could not find container \"cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86\": container with ID starting with cbcd39f1f0eff4270a9d020eefa23c5b25eea36ff79ddbb50c5864b3501acc86 not found: ID does not exist"
Sep 11 12:32:32.163556 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.163531    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"]
Sep 11 12:32:32.168004 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.167982    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-82ead5-679449cfd9-lhrdl"]
Sep 11 12:32:32.177122 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.177091    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"]
Sep 11 12:32:32.179271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:32.179252    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82ead5-67ffd4886b-xtq8j"]
Sep 11 12:32:33.113414 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:33.113366    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a96639c-eca1-489b-9981-acdae271c5b0" path="/var/lib/kubelet/pods/3a96639c-eca1-489b-9981-acdae271c5b0/volumes"
Sep 11 12:32:33.113805 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:33.113789    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2738a91-74d8-40e4-bcff-de060f2100f7" path="/var/lib/kubelet/pods/b2738a91-74d8-40e4-bcff-de060f2100f7/volumes"
Sep 11 12:32:38.189359 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:38.189328    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:38.300075 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:38.300032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:38.488329 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:38.488231    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:38.978340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:38.978304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:39.228503 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:39.228375    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:39.409079 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:39.409049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:40.424914 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:40.424867    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:40.523975 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:40.523939    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:40.627908 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:40.627877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:43.497707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:43.497675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:43.953789 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:43.953754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:44.738896 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:44.738856    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:47.070941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:47.070910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:47.305765 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:47.305725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:47.405360 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:47.405321    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:47.705288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:47.705196    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:53.862091 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:53.862051    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:53.964122 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:53.964085    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:32:57.004844 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:32:57.004812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:02.299822 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.299777    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"]
Sep 11 12:33:02.300542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.300519    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2738a91-74d8-40e4-bcff-de060f2100f7" containerName="maas-api"
Sep 11 12:33:02.300604 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.300546    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2738a91-74d8-40e4-bcff-de060f2100f7" containerName="maas-api"
Sep 11 12:33:02.300604 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.300582    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a96639c-eca1-489b-9981-acdae271c5b0" containerName="payload-pre-processing"
Sep 11 12:33:02.300604 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.300592    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a96639c-eca1-489b-9981-acdae271c5b0" containerName="payload-pre-processing"
Sep 11 12:33:02.300719 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.300706    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2738a91-74d8-40e4-bcff-de060f2100f7" containerName="maas-api"
Sep 11 12:33:02.300754 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.300724    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a96639c-eca1-489b-9981-acdae271c5b0" containerName="payload-pre-processing"
Sep 11 12:33:02.304518 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.304499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.306996 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.306970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e961b4\"/\"default-dockercfg-xrnz4\""
Sep 11 12:33:02.307845 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.307826    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e961b4\"/\"openshift-service-ca.crt\""
Sep 11 12:33:02.307948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.307873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e961b4\"/\"noten-model-e961b4-kserve-self-signed-certs\""
Sep 11 12:33:02.307948 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.307885    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e961b4\"/\"kube-root-ca.crt\""
Sep 11 12:33:02.313287 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.313261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"]
Sep 11 12:33:02.328685 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.328657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:02.393129 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.393098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-home\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.393297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.393134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-dshm\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.393297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.393238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-tmp-dir\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.393297 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.393270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2840971-7e65-4fa4-b6f5-354137afc521-tls-certs\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.393433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.393314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-model-cache\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.393433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.393338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbjq\" (UniqueName: \"kubernetes.io/projected/f2840971-7e65-4fa4-b6f5-354137afc521-kube-api-access-4tbjq\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-home\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-dshm\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-tmp-dir\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2840971-7e65-4fa4-b6f5-354137afc521-tls-certs\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-model-cache\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494436 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbjq\" (UniqueName: \"kubernetes.io/projected/f2840971-7e65-4fa4-b6f5-354137afc521-kube-api-access-4tbjq\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494668 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-home\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494668 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-tmp-dir\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.494748 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.494698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-model-cache\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.496652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.496622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-dshm\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.496764 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.496715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2840971-7e65-4fa4-b6f5-354137afc521-tls-certs\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.502588 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.502564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbjq\" (UniqueName: \"kubernetes.io/projected/f2840971-7e65-4fa4-b6f5-354137afc521-kube-api-access-4tbjq\") pod \"noten-model-e961b4-kserve-545fcdfcd7-jvdg7\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") " pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.617149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.617112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:02.891008 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:02.890981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"]
Sep 11 12:33:02.892361 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:33:02.892333    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2840971_7e65_4fa4_b6f5_354137afc521.slice/crio-91932b5908b14fefa512fab22501a86b1ad86ceb0aa32bc5b0d835f51de22024 WatchSource:0}: Error finding container 91932b5908b14fefa512fab22501a86b1ad86ceb0aa32bc5b0d835f51de22024: Status 404 returned error can't find the container with id 91932b5908b14fefa512fab22501a86b1ad86ceb0aa32bc5b0d835f51de22024
Sep 11 12:33:03.308821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:03.308719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7" event={"ID":"f2840971-7e65-4fa4-b6f5-354137afc521","Type":"ContainerStarted","Data":"606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea"}
Sep 11 12:33:03.308821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:03.308762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7" event={"ID":"f2840971-7e65-4fa4-b6f5-354137afc521","Type":"ContainerStarted","Data":"91932b5908b14fefa512fab22501a86b1ad86ceb0aa32bc5b0d835f51de22024"}
Sep 11 12:33:03.328353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:03.328306    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7" podStartSLOduration=1.328291023 podStartE2EDuration="1.328291023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:03.326950036 +0000 UTC m=+1118.717487375" watchObservedRunningTime="2026-09-11 12:33:03.328291023 +0000 UTC m=+1118.718828361"
Sep 11 12:33:04.740530 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.740487    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"]
Sep 11 12:33:04.740982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.740753    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" podUID="d1c14e28-4755-4043-b936-7de6ffd33ad8" containerName="maas-api" containerID="cri-o://b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154" gracePeriod=30
Sep 11 12:33:04.765986 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.765948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:04.844070 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.844029    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:04.902274 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.902238    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:04.967167 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.967132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:04.994942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:04.994882    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:33:05.026731 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.026702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:05.121047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.121018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls\") pod \"d1c14e28-4755-4043-b936-7de6ffd33ad8\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") "
Sep 11 12:33:05.121219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.121064    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14e28-4755-4043-b936-7de6ffd33ad8-openshift-service-ca\") pod \"d1c14e28-4755-4043-b936-7de6ffd33ad8\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") "
Sep 11 12:33:05.121219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.121188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/d1c14e28-4755-4043-b936-7de6ffd33ad8-kube-api-access-dqfw8\") pod \"d1c14e28-4755-4043-b936-7de6ffd33ad8\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") "
Sep 11 12:33:05.121341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.121240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-metrics-certs\") pod \"d1c14e28-4755-4043-b936-7de6ffd33ad8\" (UID: \"d1c14e28-4755-4043-b936-7de6ffd33ad8\") "
Sep 11 12:33:05.121488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.121458    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c14e28-4755-4043-b936-7de6ffd33ad8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1c14e28-4755-4043-b936-7de6ffd33ad8" (UID: "d1c14e28-4755-4043-b936-7de6ffd33ad8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:33:05.121603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.121578    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14e28-4755-4043-b936-7de6ffd33ad8-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.123373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.123345    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d1c14e28-4755-4043-b936-7de6ffd33ad8" (UID: "d1c14e28-4755-4043-b936-7de6ffd33ad8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:33:05.123513 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.123367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1c14e28-4755-4043-b936-7de6ffd33ad8" (UID: "d1c14e28-4755-4043-b936-7de6ffd33ad8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:33:05.123773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.123671    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c14e28-4755-4043-b936-7de6ffd33ad8-kube-api-access-dqfw8" (OuterVolumeSpecName: "kube-api-access-dqfw8") pod "d1c14e28-4755-4043-b936-7de6ffd33ad8" (UID: "d1c14e28-4755-4043-b936-7de6ffd33ad8"). InnerVolumeSpecName "kube-api-access-dqfw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:05.222379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.222342    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.222379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.222373    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c14e28-4755-4043-b936-7de6ffd33ad8-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.222379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.222383    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqfw8\" (UniqueName: \"kubernetes.io/projected/d1c14e28-4755-4043-b936-7de6ffd33ad8-kube-api-access-dqfw8\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.326562 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.326528    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1c14e28-4755-4043-b936-7de6ffd33ad8" containerID="b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154" exitCode=0
Sep 11 12:33:05.326728 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.326581    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"
Sep 11 12:33:05.326728 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.326619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" event={"ID":"d1c14e28-4755-4043-b936-7de6ffd33ad8","Type":"ContainerDied","Data":"b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154"}
Sep 11 12:33:05.326728 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.326693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh" event={"ID":"d1c14e28-4755-4043-b936-7de6ffd33ad8","Type":"ContainerDied","Data":"adad7505404c9325fe59501eaa58ef65435b46fbde488c2398d7112abda90ed9"}
Sep 11 12:33:05.326728 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.326715    2561 scope.go:117] "RemoveContainer" containerID="b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154"
Sep 11 12:33:05.336236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.336221    2561 scope.go:117] "RemoveContainer" containerID="b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154"
Sep 11 12:33:05.336515 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:05.336496    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154\": container with ID starting with b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154 not found: ID does not exist" containerID="b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154"
Sep 11 12:33:05.336565 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.336522    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154"} err="failed to get container status \"b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154\": rpc error: code = NotFound desc = could not find container \"b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154\": container with ID starting with b8436bd5896db4a85205ba00ce84b128ee791e987953d30ca8b205a2582c4154 not found: ID does not exist"
Sep 11 12:33:05.349738 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.349717    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"]
Sep 11 12:33:05.353775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.353751    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad618118-77bf94db8c-xw2jh"]
Sep 11 12:33:05.401959 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:05.401921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:07.107347 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:07.107313    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c14e28-4755-4043-b936-7de6ffd33ad8" path="/var/lib/kubelet/pods/d1c14e28-4755-4043-b936-7de6ffd33ad8/volumes"
Sep 11 12:33:11.191269 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:11.191228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:11.256118 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:11.256083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:11.437720 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:11.437686    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:11.743543 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:11.743508    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:11.841685 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:11.841638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:11.952709 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:11.952677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:12.617717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:12.617679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:12.618163 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:12.617734    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:12.631089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:12.631057    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:13.382773 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:13.382744    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:17.814544 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:17.814504    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:17.868831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:17.868796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:17.961643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:17.961596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:18.062645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:18.062608    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:18.918649 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:18.918614    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:22.829367 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:22.829331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:24.083542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:24.083506    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"]
Sep 11 12:33:24.095306 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:24.095273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:24.212165 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:24.212120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:24.337088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:24.337002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:24.438126 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:24.438081    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:25.346234 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:25.346202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:25.448284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:25.448244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:25.533577 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:25.533476    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:25.599926 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:25.599888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:25.794628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:25.794471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:27.271392 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:27.271359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:28.400596 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:28.400559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:28.498060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:28.498027    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:29.520858 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.520821    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"]
Sep 11 12:33:29.521333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.521266    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c14e28-4755-4043-b936-7de6ffd33ad8" containerName="maas-api"
Sep 11 12:33:29.521333 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.521279    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c14e28-4755-4043-b936-7de6ffd33ad8" containerName="maas-api"
Sep 11 12:33:29.521504 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.521361    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c14e28-4755-4043-b936-7de6ffd33ad8" containerName="maas-api"
Sep 11 12:33:29.524667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.524645    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.527047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.527024    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-44f372-serving-cert\""
Sep 11 12:33:29.534353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.534327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"]
Sep 11 12:33:29.554373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.554335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:29.623344 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.623273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:29.658141 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.658094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfrj\" (UniqueName: \"kubernetes.io/projected/be3b6a74-ae75-403e-8734-bc7922a704f6-kube-api-access-vkfrj\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.658336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.658212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.658336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.658293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.658449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.658369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b6a74-ae75-403e-8734-bc7922a704f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.758938 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.758899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfrj\" (UniqueName: \"kubernetes.io/projected/be3b6a74-ae75-403e-8734-bc7922a704f6-kube-api-access-vkfrj\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.759137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.758980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.759137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.759024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.759137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.759075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b6a74-ae75-403e-8734-bc7922a704f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.759314 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:29.759142    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-serving-cert: secret "maas-api-e2e-shared-a-w2-44f372-serving-cert" not found
Sep 11 12:33:29.759314 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:29.759234    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls podName:be3b6a74-ae75-403e-8734-bc7922a704f6 nodeName:}" failed. No retries permitted until 2026-09-11 12:33:30.259212429 +0000 UTC m=+1145.649749762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls") pod "maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" (UID: "be3b6a74-ae75-403e-8734-bc7922a704f6") : secret "maas-api-e2e-shared-a-w2-44f372-serving-cert" not found
Sep 11 12:33:29.759774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.759754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b6a74-ae75-403e-8734-bc7922a704f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.761755 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.761735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:29.767185 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:29.767156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfrj\" (UniqueName: \"kubernetes.io/projected/be3b6a74-ae75-403e-8734-bc7922a704f6-kube-api-access-vkfrj\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:30.015340 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.015286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:30.115063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.115026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:30.132241 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.132210    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"]
Sep 11 12:33:30.136041 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.136020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.138134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.138115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-44f372\""
Sep 11 12:33:30.138236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.138115    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-44f372-dockercfg-nggr9\""
Sep 11 12:33:30.146246 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.146223    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"]
Sep 11 12:33:30.189564 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.189523    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:30.270472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.270375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.270617 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.270549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:30.270617 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.270604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxdz\" (UniqueName: \"kubernetes.io/projected/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-kube-api-access-szxdz\") pod \"payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.272996 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.272968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:30.371428 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.371359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.371610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.371465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxdz\" (UniqueName: \"kubernetes.io/projected/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-kube-api-access-szxdz\") pod \"payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.372112 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.372090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.379936 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.379903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxdz\" (UniqueName: \"kubernetes.io/projected/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-kube-api-access-szxdz\") pod \"payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.440461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.440419    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:30.449470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.449445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:30.721917 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.721879    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"]
Sep 11 12:33:30.724364 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:33:30.724337    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3b6a74_ae75_403e_8734_bc7922a704f6.slice/crio-666624d6cc8a1aa35df061373c3d8e2c76edfe230a98f05408b5024e9f738342 WatchSource:0}: Error finding container 666624d6cc8a1aa35df061373c3d8e2c76edfe230a98f05408b5024e9f738342: Status 404 returned error can't find the container with id 666624d6cc8a1aa35df061373c3d8e2c76edfe230a98f05408b5024e9f738342
Sep 11 12:33:30.724799 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:30.724721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"]
Sep 11 12:33:30.726378 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:33:30.726356    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4008a2b4_2a13_4f81_a5d2_a3c8d6f4b637.slice/crio-84d0fe3f02d0af8eca21347a48228d5300c2cc10d46f30bf76e53dcaf7f58f9e WatchSource:0}: Error finding container 84d0fe3f02d0af8eca21347a48228d5300c2cc10d46f30bf76e53dcaf7f58f9e: Status 404 returned error can't find the container with id 84d0fe3f02d0af8eca21347a48228d5300c2cc10d46f30bf76e53dcaf7f58f9e
Sep 11 12:33:31.482968 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.482876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6" event={"ID":"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637","Type":"ContainerStarted","Data":"159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d"}
Sep 11 12:33:31.482968 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.482939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6" event={"ID":"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637","Type":"ContainerStarted","Data":"84d0fe3f02d0af8eca21347a48228d5300c2cc10d46f30bf76e53dcaf7f58f9e"}
Sep 11 12:33:31.483192 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.483080    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:31.484644 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.484619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" event={"ID":"be3b6a74-ae75-403e-8734-bc7922a704f6","Type":"ContainerStarted","Data":"f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7"}
Sep 11 12:33:31.484787 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.484658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" event={"ID":"be3b6a74-ae75-403e-8734-bc7922a704f6","Type":"ContainerStarted","Data":"666624d6cc8a1aa35df061373c3d8e2c76edfe230a98f05408b5024e9f738342"}
Sep 11 12:33:31.484787 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.484704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:31.500774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.500731    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6" podStartSLOduration=1.500717633 podStartE2EDuration="1.500717633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:31.499207137 +0000 UTC m=+1146.889744479" watchObservedRunningTime="2026-09-11 12:33:31.500717633 +0000 UTC m=+1146.891255000"
Sep 11 12:33:31.517237 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:31.517199    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" podStartSLOduration=2.129781512 podStartE2EDuration="2.51718626s" totalImagesPullingTime="387.404748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:31.515139533 +0000 UTC m=+1146.905676875" watchObservedRunningTime="2026-09-11 12:33:31.51718626 +0000 UTC m=+1146.907723635"
Sep 11 12:33:37.088103 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:37.088069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:37.495289 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:37.495210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:33:39.104443 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.104384    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7" podUID="f2840971-7e65-4fa4-b6f5-354137afc521" containerName="main" containerID="cri-o://606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea" gracePeriod=45
Sep 11 12:33:39.359957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.359896    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:39.465226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tbjq\" (UniqueName: \"kubernetes.io/projected/f2840971-7e65-4fa4-b6f5-354137afc521-kube-api-access-4tbjq\") pod \"f2840971-7e65-4fa4-b6f5-354137afc521\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") "
Sep 11 12:33:39.465389 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-model-cache\") pod \"f2840971-7e65-4fa4-b6f5-354137afc521\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") "
Sep 11 12:33:39.465389 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465373    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-tmp-dir\") pod \"f2840971-7e65-4fa4-b6f5-354137afc521\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") "
Sep 11 12:33:39.465554 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-home\") pod \"f2840971-7e65-4fa4-b6f5-354137afc521\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") "
Sep 11 12:33:39.465554 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-dshm\") pod \"f2840971-7e65-4fa4-b6f5-354137afc521\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") "
Sep 11 12:33:39.465554 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2840971-7e65-4fa4-b6f5-354137afc521-tls-certs\") pod \"f2840971-7e65-4fa4-b6f5-354137afc521\" (UID: \"f2840971-7e65-4fa4-b6f5-354137afc521\") "
Sep 11 12:33:39.465710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465573    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2840971-7e65-4fa4-b6f5-354137afc521" (UID: "f2840971-7e65-4fa4-b6f5-354137afc521"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:39.465710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2840971-7e65-4fa4-b6f5-354137afc521" (UID: "f2840971-7e65-4fa4-b6f5-354137afc521"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:39.465809 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465707    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-home" (OuterVolumeSpecName: "home") pod "f2840971-7e65-4fa4-b6f5-354137afc521" (UID: "f2840971-7e65-4fa4-b6f5-354137afc521"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:39.465883 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465866    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-tmp-dir\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:39.465937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465888    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-home\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:39.465937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.465901    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-model-cache\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:39.467516 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.467496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-dshm" (OuterVolumeSpecName: "dshm") pod "f2840971-7e65-4fa4-b6f5-354137afc521" (UID: "f2840971-7e65-4fa4-b6f5-354137afc521"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:39.467516 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.467509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2840971-7e65-4fa4-b6f5-354137afc521-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2840971-7e65-4fa4-b6f5-354137afc521" (UID: "f2840971-7e65-4fa4-b6f5-354137afc521"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:33:39.467675 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.467512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2840971-7e65-4fa4-b6f5-354137afc521-kube-api-access-4tbjq" (OuterVolumeSpecName: "kube-api-access-4tbjq") pod "f2840971-7e65-4fa4-b6f5-354137afc521" (UID: "f2840971-7e65-4fa4-b6f5-354137afc521"). InnerVolumeSpecName "kube-api-access-4tbjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:39.527863 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.527836    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2840971-7e65-4fa4-b6f5-354137afc521" containerID="606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea" exitCode=0
Sep 11 12:33:39.527961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.527881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7" event={"ID":"f2840971-7e65-4fa4-b6f5-354137afc521","Type":"ContainerDied","Data":"606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea"}
Sep 11 12:33:39.527961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.527885    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"
Sep 11 12:33:39.527961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.527911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7" event={"ID":"f2840971-7e65-4fa4-b6f5-354137afc521","Type":"ContainerDied","Data":"91932b5908b14fefa512fab22501a86b1ad86ceb0aa32bc5b0d835f51de22024"}
Sep 11 12:33:39.527961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.527927    2561 scope.go:117] "RemoveContainer" containerID="606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea"
Sep 11 12:33:39.537460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.537440    2561 scope.go:117] "RemoveContainer" containerID="606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea"
Sep 11 12:33:39.537781 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:39.537763    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea\": container with ID starting with 606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea not found: ID does not exist" containerID="606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea"
Sep 11 12:33:39.537841 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.537790    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea"} err="failed to get container status \"606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea\": rpc error: code = NotFound desc = could not find container \"606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea\": container with ID starting with 606f1b7b9b755859baffcc1f43ddf8438bdf023ad3e7962d22e974f3e91929ea not found: ID does not exist"
Sep 11 12:33:39.549437 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.549394    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"]
Sep 11 12:33:39.555290 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.555268    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e961b4/noten-model-e961b4-kserve-545fcdfcd7-jvdg7"]
Sep 11 12:33:39.566545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.566528    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2840971-7e65-4fa4-b6f5-354137afc521-dshm\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:39.566609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.566548    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2840971-7e65-4fa4-b6f5-354137afc521-tls-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:39.566609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:39.566560    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbjq\" (UniqueName: \"kubernetes.io/projected/f2840971-7e65-4fa4-b6f5-354137afc521-kube-api-access-4tbjq\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.106886 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:41.106854    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2840971-7e65-4fa4-b6f5-354137afc521" path="/var/lib/kubelet/pods/f2840971-7e65-4fa4-b6f5-354137afc521/volumes"
Sep 11 12:33:41.393313 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:41.393227    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:41.619803 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:41.619757    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:42.492277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:42.492244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:33:45.664514 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:45.664478    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:50.418711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:50.418672    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:51.566758 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.566723    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"]
Sep 11 12:33:51.567152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.566983    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" podUID="4d901b4a-ee48-41a7-be49-52a9f05b5c21" containerName="maas-api" containerID="cri-o://faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984" gracePeriod=30
Sep 11 12:33:51.590727 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.590694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:51.596205 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.596179    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"]
Sep 11 12:33:51.599753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.599730    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38a724-29818830-zdf8d"]
Sep 11 12:33:51.632086 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.632057    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"]
Sep 11 12:33:51.632809 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.632666    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5" podUID="58c78d0a-4ed5-4578-ba06-2c85e73edd73" containerName="payload-pre-processing" containerID="cri-o://f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02" gracePeriod=30
Sep 11 12:33:51.692262 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.692223    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:51.791758 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.791723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:51.894657 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.894572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:51.919625 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.919590    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:33:51.935101 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.935081    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:33:51.961521 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.961491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:51.987653 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.987627    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c78d0a-4ed5-4578-ba06-2c85e73edd73-plugins-config-volume\") pod \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") "
Sep 11 12:33:51.987774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.987687    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d901b4a-ee48-41a7-be49-52a9f05b5c21-openshift-service-ca\") pod \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") "
Sep 11 12:33:51.987774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.987742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-metrics-certs\") pod \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") "
Sep 11 12:33:51.987881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.987866    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") pod \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") "
Sep 11 12:33:51.987917 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.987897    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqvvl\" (UniqueName: \"kubernetes.io/projected/4d901b4a-ee48-41a7-be49-52a9f05b5c21-kube-api-access-rqvvl\") pod \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\" (UID: \"4d901b4a-ee48-41a7-be49-52a9f05b5c21\") "
Sep 11 12:33:51.987963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.987943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27xfc\" (UniqueName: \"kubernetes.io/projected/58c78d0a-4ed5-4578-ba06-2c85e73edd73-kube-api-access-27xfc\") pod \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\" (UID: \"58c78d0a-4ed5-4578-ba06-2c85e73edd73\") "
Sep 11 12:33:51.988085 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.988055    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d901b4a-ee48-41a7-be49-52a9f05b5c21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d901b4a-ee48-41a7-be49-52a9f05b5c21" (UID: "4d901b4a-ee48-41a7-be49-52a9f05b5c21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:33:51.988157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.988067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c78d0a-4ed5-4578-ba06-2c85e73edd73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58c78d0a-4ed5-4578-ba06-2c85e73edd73" (UID: "58c78d0a-4ed5-4578-ba06-2c85e73edd73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:33:51.988356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.988331    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c78d0a-4ed5-4578-ba06-2c85e73edd73-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:51.988356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.988356    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d901b4a-ee48-41a7-be49-52a9f05b5c21-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:51.990211 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.990183    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d901b4a-ee48-41a7-be49-52a9f05b5c21" (UID: "4d901b4a-ee48-41a7-be49-52a9f05b5c21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:33:51.990485 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.990456    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c78d0a-4ed5-4578-ba06-2c85e73edd73-kube-api-access-27xfc" (OuterVolumeSpecName: "kube-api-access-27xfc") pod "58c78d0a-4ed5-4578-ba06-2c85e73edd73" (UID: "58c78d0a-4ed5-4578-ba06-2c85e73edd73"). InnerVolumeSpecName "kube-api-access-27xfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:51.990593 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.990481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d901b4a-ee48-41a7-be49-52a9f05b5c21-kube-api-access-rqvvl" (OuterVolumeSpecName: "kube-api-access-rqvvl") pod "4d901b4a-ee48-41a7-be49-52a9f05b5c21" (UID: "4d901b4a-ee48-41a7-be49-52a9f05b5c21"). InnerVolumeSpecName "kube-api-access-rqvvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:51.990593 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:51.990568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d901b4a-ee48-41a7-be49-52a9f05b5c21" (UID: "4d901b4a-ee48-41a7-be49-52a9f05b5c21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:33:52.089273 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.089245    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:52.089273 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.089270    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d901b4a-ee48-41a7-be49-52a9f05b5c21-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:52.089477 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.089284    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqvvl\" (UniqueName: \"kubernetes.io/projected/4d901b4a-ee48-41a7-be49-52a9f05b5c21-kube-api-access-rqvvl\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:52.089477 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.089297    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27xfc\" (UniqueName: \"kubernetes.io/projected/58c78d0a-4ed5-4578-ba06-2c85e73edd73-kube-api-access-27xfc\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:33:52.303826 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.303740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:52.598337 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.598301    2561 generic.go:356] "Generic (PLEG): container finished" podID="58c78d0a-4ed5-4578-ba06-2c85e73edd73" containerID="f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02" exitCode=0
Sep 11 12:33:52.598743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.598342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"
Sep 11 12:33:52.598743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.598390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5" event={"ID":"58c78d0a-4ed5-4578-ba06-2c85e73edd73","Type":"ContainerDied","Data":"f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02"}
Sep 11 12:33:52.598743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.598487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5" event={"ID":"58c78d0a-4ed5-4578-ba06-2c85e73edd73","Type":"ContainerDied","Data":"0440d5cb2ef9f3c0876f3642c60a701115f4185bef7d57dcbcc6c1a768501568"}
Sep 11 12:33:52.598743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.598523    2561 scope.go:117] "RemoveContainer" containerID="f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02"
Sep 11 12:33:52.599881 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.599858    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d901b4a-ee48-41a7-be49-52a9f05b5c21" containerID="faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984" exitCode=0
Sep 11 12:33:52.599942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.599891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" event={"ID":"4d901b4a-ee48-41a7-be49-52a9f05b5c21","Type":"ContainerDied","Data":"faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984"}
Sep 11 12:33:52.599942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.599912    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"
Sep 11 12:33:52.599942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.599925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr" event={"ID":"4d901b4a-ee48-41a7-be49-52a9f05b5c21","Type":"ContainerDied","Data":"1cad0e5fd88e05c8dfc6d8ed99ef30d1583d04a0272bcb5510f6c180a77417d4"}
Sep 11 12:33:52.606939 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.606922    2561 scope.go:117] "RemoveContainer" containerID="f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02"
Sep 11 12:33:52.607201 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:52.607180    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02\": container with ID starting with f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02 not found: ID does not exist" containerID="f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02"
Sep 11 12:33:52.607277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.607206    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02"} err="failed to get container status \"f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02\": rpc error: code = NotFound desc = could not find container \"f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02\": container with ID starting with f5b0b87a5db7d5b128dead1aeceea5c90d4e3df81f0bfc481f0d9ce7900b3d02 not found: ID does not exist"
Sep 11 12:33:52.607277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.607229    2561 scope.go:117] "RemoveContainer" containerID="faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984"
Sep 11 12:33:52.616570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.616527    2561 scope.go:117] "RemoveContainer" containerID="faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984"
Sep 11 12:33:52.617730 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:52.617696    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984\": container with ID starting with faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984 not found: ID does not exist" containerID="faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984"
Sep 11 12:33:52.617828 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.617742    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984"} err="failed to get container status \"faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984\": rpc error: code = NotFound desc = could not find container \"faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984\": container with ID starting with faeff26072b9770d95eeb220f02a41c97cfd9b453afcd8482854e305569c1984 not found: ID does not exist"
Sep 11 12:33:52.629570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.629544    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"]
Sep 11 12:33:52.634197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.634172    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-38a724-59bb5bc958-6rnk5"]
Sep 11 12:33:52.647986 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.647965    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"]
Sep 11 12:33:52.654159 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:52.654138    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-38a724-67fc6bc5b-9bgxr"]
Sep 11 12:33:53.106862 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.106828    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d901b4a-ee48-41a7-be49-52a9f05b5c21" path="/var/lib/kubelet/pods/4d901b4a-ee48-41a7-be49-52a9f05b5c21/volumes"
Sep 11 12:33:53.107228 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.107215    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c78d0a-4ed5-4578-ba06-2c85e73edd73" path="/var/lib/kubelet/pods/58c78d0a-4ed5-4578-ba06-2c85e73edd73/volumes"
Sep 11 12:33:53.107486 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.107468    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f3c3c6-7871-4057-ad56-a2604731e258" path="/var/lib/kubelet/pods/59f3c3c6-7871-4057-ad56-a2604731e258/volumes"
Sep 11 12:33:53.320935 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.320897    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:53.423149 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.423054    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:53.643999 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.643959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:53.911499 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.911466    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"]
Sep 11 12:33:53.912095 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912081    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2840971-7e65-4fa4-b6f5-354137afc521" containerName="main"
Sep 11 12:33:53.912152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912099    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2840971-7e65-4fa4-b6f5-354137afc521" containerName="main"
Sep 11 12:33:53.912152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912124    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c78d0a-4ed5-4578-ba06-2c85e73edd73" containerName="payload-pre-processing"
Sep 11 12:33:53.912152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912134    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c78d0a-4ed5-4578-ba06-2c85e73edd73" containerName="payload-pre-processing"
Sep 11 12:33:53.912152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912151    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d901b4a-ee48-41a7-be49-52a9f05b5c21" containerName="maas-api"
Sep 11 12:33:53.912288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912159    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d901b4a-ee48-41a7-be49-52a9f05b5c21" containerName="maas-api"
Sep 11 12:33:53.912288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912256    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d901b4a-ee48-41a7-be49-52a9f05b5c21" containerName="maas-api"
Sep 11 12:33:53.912288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912268    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c78d0a-4ed5-4578-ba06-2c85e73edd73" containerName="payload-pre-processing"
Sep 11 12:33:53.912288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.912280    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2840971-7e65-4fa4-b6f5-354137afc521" containerName="main"
Sep 11 12:33:53.916023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.916001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:53.918440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.918417    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a50cf9-openshift-default-dockercfg-c8ncm\""
Sep 11 12:33:53.928375 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.928352    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"]
Sep 11 12:33:53.944168 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:53.944035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:54.006571 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-credential-socket\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-certs\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-kube-api-access-7lbqw\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-token\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-socket\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-data\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006974 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-envoy\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.006974 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.006816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-podinfo\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.107742 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-envoy\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.107892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-podinfo\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.107892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-credential-socket\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.107892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-certs\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.107892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-kube-api-access-7lbqw\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-token\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-socket\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.107999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-data\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108268 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.108239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-credential-socket\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108323 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.108273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-certs\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108377 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.108318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-data\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.108708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-socket\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.108876 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.108855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.110221 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.110200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-envoy\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.110351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.110335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-podinfo\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.115892 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.115868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-token\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.116137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.116118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-kube-api-access-7lbqw\") pod \"e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") " pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.229883 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.229803    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:54.328838 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.328777    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:54.513845 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.513816    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"]
Sep 11 12:33:54.515388 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:33:54.515353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3548b9a1_9db3_4b02_a1ce_34064f2407fc.slice/crio-6c24a0d9b5ed43afbba605d54be04287b4f204113929486dc4df161362446873 WatchSource:0}: Error finding container 6c24a0d9b5ed43afbba605d54be04287b4f204113929486dc4df161362446873: Status 404 returned error can't find the container with id 6c24a0d9b5ed43afbba605d54be04287b4f204113929486dc4df161362446873
Sep 11 12:33:54.518117 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.518085    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:33:54.518193 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.518150    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:33:54.518193 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.518181    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:33:54.615471 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.615433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" event={"ID":"3548b9a1-9db3-4b02-a1ce-34064f2407fc","Type":"ContainerStarted","Data":"0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"}
Sep 11 12:33:54.615610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.615500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" event={"ID":"3548b9a1-9db3-4b02-a1ce-34064f2407fc","Type":"ContainerStarted","Data":"6c24a0d9b5ed43afbba605d54be04287b4f204113929486dc4df161362446873"}
Sep 11 12:33:54.637469 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:54.637389    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podStartSLOduration=1.637369071 podStartE2EDuration="1.637369071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:54.633917783 +0000 UTC m=+1170.024455127" watchObservedRunningTime="2026-09-11 12:33:54.637369071 +0000 UTC m=+1170.027906415"
Sep 11 12:33:55.230213 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:55.230175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:55.234848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:55.234823    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:55.621595 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:55.621569    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:55.622435 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:55.622391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:33:56.004482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:56.004373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:56.066779 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:56.066749    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:56.519428 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:56.519370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:56.613125 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:56.613089    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:57.803318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.803285    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"]
Sep 11 12:33:57.807473 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.807444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.809897 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.809879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a50cf9-serving-cert\""
Sep 11 12:33:57.819676 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.819651    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"]
Sep 11 12:33:57.844926 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.844883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:57.845245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.845222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.845319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.845310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dk55\" (UniqueName: \"kubernetes.io/projected/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-kube-api-access-8dk55\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.845384 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.845355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.845384 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.845377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.944820 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.944786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:57.945934 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.945909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.946017 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.945943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.946017 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.946012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.946123 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:57.946074    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-serving-cert: secret "maas-api-e2e-shared-b-w2-a50cf9-serving-cert" not found
Sep 11 12:33:57.946177 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.946078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dk55\" (UniqueName: \"kubernetes.io/projected/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-kube-api-access-8dk55\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.946177 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:33:57.946143    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls podName:35380b8a-2c03-4cfc-9e92-e5ea9da0f126 nodeName:}" failed. No retries permitted until 2026-09-11 12:33:58.446121065 +0000 UTC m=+1173.836658403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" (UID: "35380b8a-2c03-4cfc-9e92-e5ea9da0f126") : secret "maas-api-e2e-shared-b-w2-a50cf9-serving-cert" not found
Sep 11 12:33:57.946782 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.946759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.948383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.948362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:57.954140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:57.954118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dk55\" (UniqueName: \"kubernetes.io/projected/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-kube-api-access-8dk55\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:58.316135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.316094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:58.451195 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.451163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:58.453684 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.453651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:58.476926 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.476896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:58.578527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.578496    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:58.720084 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.720012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:58.998717 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:58.998687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"]
Sep 11 12:33:59.001223 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:33:59.001195    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35380b8a_2c03_4cfc_9e92_e5ea9da0f126.slice/crio-71b24da0d45dadbc18e0fc7932454acb69345a0d3cc3f4fa7335a7dc572aed2d WatchSource:0}: Error finding container 71b24da0d45dadbc18e0fc7932454acb69345a0d3cc3f4fa7335a7dc572aed2d: Status 404 returned error can't find the container with id 71b24da0d45dadbc18e0fc7932454acb69345a0d3cc3f4fa7335a7dc572aed2d
Sep 11 12:33:59.035777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.035749    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:59.135473 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.135391    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:59.301937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.301848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:33:59.648349 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.648317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" event={"ID":"35380b8a-2c03-4cfc-9e92-e5ea9da0f126","Type":"ContainerStarted","Data":"21228f43449f3e0377b0ae18ae2c3186443b5b63ff332d43a99c778aef600347"}
Sep 11 12:33:59.648534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.648372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" event={"ID":"35380b8a-2c03-4cfc-9e92-e5ea9da0f126","Type":"ContainerStarted","Data":"71b24da0d45dadbc18e0fc7932454acb69345a0d3cc3f4fa7335a7dc572aed2d"}
Sep 11 12:33:59.648534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.648442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:33:59.666236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.665990    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" podStartSLOduration=2.229220225 podStartE2EDuration="2.665969162s" totalImagesPullingTime="436.748937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:59.664211623 +0000 UTC m=+1175.054748967" watchObservedRunningTime="2026-09-11 12:33:59.665969162 +0000 UTC m=+1175.056506506"
Sep 11 12:33:59.786830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:33:59.786794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:05.659380 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:05.659349    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:34:06.837037 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:06.837002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:07.985422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:07.985367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:10.241635 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.241584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:10.274643 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.274611    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"]
Sep 11 12:34:10.274934 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.274901    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v" podUID="230ec79b-160b-4621-ae0c-17e368f30bb4" containerName="payload-processing" containerID="cri-o://9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818" gracePeriod=30
Sep 11 12:34:10.282240 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.282086    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"]
Sep 11 12:34:10.282382 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.282308    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7" podUID="9368df61-f662-4d3d-9ae4-0adaa5334eff" containerName="payload-pre-processing" containerID="cri-o://ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449" gracePeriod=30
Sep 11 12:34:10.318486 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.318445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:10.380592 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.380556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:10.498391 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.498311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:10.565929 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.565901    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:34:10.568633 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.568610    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:34:10.569020 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.569000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:10.669963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.669920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgrzq\" (UniqueName: \"kubernetes.io/projected/9368df61-f662-4d3d-9ae4-0adaa5334eff-kube-api-access-dgrzq\") pod \"9368df61-f662-4d3d-9ae4-0adaa5334eff\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") "
Sep 11 12:34:10.669963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.669964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368df61-f662-4d3d-9ae4-0adaa5334eff-plugins-config-volume\") pod \"9368df61-f662-4d3d-9ae4-0adaa5334eff\" (UID: \"9368df61-f662-4d3d-9ae4-0adaa5334eff\") "
Sep 11 12:34:10.670203 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.670021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmsxp\" (UniqueName: \"kubernetes.io/projected/230ec79b-160b-4621-ae0c-17e368f30bb4-kube-api-access-wmsxp\") pod \"230ec79b-160b-4621-ae0c-17e368f30bb4\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") "
Sep 11 12:34:10.670203 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.670077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230ec79b-160b-4621-ae0c-17e368f30bb4-plugins-config-volume\") pod \"230ec79b-160b-4621-ae0c-17e368f30bb4\" (UID: \"230ec79b-160b-4621-ae0c-17e368f30bb4\") "
Sep 11 12:34:10.670362 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.670329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9368df61-f662-4d3d-9ae4-0adaa5334eff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9368df61-f662-4d3d-9ae4-0adaa5334eff" (UID: "9368df61-f662-4d3d-9ae4-0adaa5334eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:34:10.670545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.670494    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230ec79b-160b-4621-ae0c-17e368f30bb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "230ec79b-160b-4621-ae0c-17e368f30bb4" (UID: "230ec79b-160b-4621-ae0c-17e368f30bb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:34:10.670545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.670504    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9368df61-f662-4d3d-9ae4-0adaa5334eff-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.672021 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.671999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9368df61-f662-4d3d-9ae4-0adaa5334eff-kube-api-access-dgrzq" (OuterVolumeSpecName: "kube-api-access-dgrzq") pod "9368df61-f662-4d3d-9ae4-0adaa5334eff" (UID: "9368df61-f662-4d3d-9ae4-0adaa5334eff"). InnerVolumeSpecName "kube-api-access-dgrzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:34:10.672090 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.672075    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230ec79b-160b-4621-ae0c-17e368f30bb4-kube-api-access-wmsxp" (OuterVolumeSpecName: "kube-api-access-wmsxp") pod "230ec79b-160b-4621-ae0c-17e368f30bb4" (UID: "230ec79b-160b-4621-ae0c-17e368f30bb4"). InnerVolumeSpecName "kube-api-access-wmsxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:34:10.708257 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.708229    2561 generic.go:356] "Generic (PLEG): container finished" podID="230ec79b-160b-4621-ae0c-17e368f30bb4" containerID="9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818" exitCode=0
Sep 11 12:34:10.708386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.708285    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"
Sep 11 12:34:10.708386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.708315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v" event={"ID":"230ec79b-160b-4621-ae0c-17e368f30bb4","Type":"ContainerDied","Data":"9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818"}
Sep 11 12:34:10.708542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.708389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v" event={"ID":"230ec79b-160b-4621-ae0c-17e368f30bb4","Type":"ContainerDied","Data":"e8c8a8d5964cb5bd28818c9e297b3e79ab88ad68282f350be9f4b9593e0785c2"}
Sep 11 12:34:10.708542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.708445    2561 scope.go:117] "RemoveContainer" containerID="9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818"
Sep 11 12:34:10.709712 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.709686    2561 generic.go:356] "Generic (PLEG): container finished" podID="9368df61-f662-4d3d-9ae4-0adaa5334eff" containerID="ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449" exitCode=0
Sep 11 12:34:10.709819 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.709723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7" event={"ID":"9368df61-f662-4d3d-9ae4-0adaa5334eff","Type":"ContainerDied","Data":"ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449"}
Sep 11 12:34:10.709819 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.709748    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"
Sep 11 12:34:10.709819 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.709760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7" event={"ID":"9368df61-f662-4d3d-9ae4-0adaa5334eff","Type":"ContainerDied","Data":"76c1f6206f82cd945bd4df85e8348e21af8c02adbcf10c73e3c951585f6171a8"}
Sep 11 12:34:10.719244 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.719230    2561 scope.go:117] "RemoveContainer" containerID="9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818"
Sep 11 12:34:10.719668 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:34:10.719646    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818\": container with ID starting with 9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818 not found: ID does not exist" containerID="9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818"
Sep 11 12:34:10.719770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.719672    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818"} err="failed to get container status \"9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818\": rpc error: code = NotFound desc = could not find container \"9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818\": container with ID starting with 9cec16671921bf1830cdac5e81b33ddc90d471eb5faf47b23315ca9342189818 not found: ID does not exist"
Sep 11 12:34:10.719770 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.719688    2561 scope.go:117] "RemoveContainer" containerID="ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449"
Sep 11 12:34:10.727598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.727575    2561 scope.go:117] "RemoveContainer" containerID="ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449"
Sep 11 12:34:10.727843 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:34:10.727826    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449\": container with ID starting with ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449 not found: ID does not exist" containerID="ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449"
Sep 11 12:34:10.727888 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.727849    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449"} err="failed to get container status \"ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449\": rpc error: code = NotFound desc = could not find container \"ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449\": container with ID starting with ddf78a6eb3741c9cc50962600731a7c2bde1a7b63504a31372c522fa51212449 not found: ID does not exist"
Sep 11 12:34:10.730951 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.730929    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"]
Sep 11 12:34:10.735182 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.735161    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-edffad-75cbdb55-f2w5v"]
Sep 11 12:34:10.744950 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.744931    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"]
Sep 11 12:34:10.751695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.751648    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-edffad-5dd4899b66-4nwb7"]
Sep 11 12:34:10.771028 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.771011    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrzq\" (UniqueName: \"kubernetes.io/projected/9368df61-f662-4d3d-9ae4-0adaa5334eff-kube-api-access-dgrzq\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.771104 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.771030    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsxp\" (UniqueName: \"kubernetes.io/projected/230ec79b-160b-4621-ae0c-17e368f30bb4-kube-api-access-wmsxp\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.771104 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.771040    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230ec79b-160b-4621-ae0c-17e368f30bb4-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.863506 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:10.863477    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:11.109143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:11.109104    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230ec79b-160b-4621-ae0c-17e368f30bb4" path="/var/lib/kubelet/pods/230ec79b-160b-4621-ae0c-17e368f30bb4/volumes"
Sep 11 12:34:11.109707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:11.109687    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9368df61-f662-4d3d-9ae4-0adaa5334eff" path="/var/lib/kubelet/pods/9368df61-f662-4d3d-9ae4-0adaa5334eff/volumes"
Sep 11 12:34:16.826349 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:16.826313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:16.882762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:16.882729    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:17.113652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:17.113558    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:17.750973 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:17.750935    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:18.048537 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.048452    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"]
Sep 11 12:34:18.049108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.049084    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9368df61-f662-4d3d-9ae4-0adaa5334eff" containerName="payload-pre-processing"
Sep 11 12:34:18.049108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.049104    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9368df61-f662-4d3d-9ae4-0adaa5334eff" containerName="payload-pre-processing"
Sep 11 12:34:18.049299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.049142    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230ec79b-160b-4621-ae0c-17e368f30bb4" containerName="payload-processing"
Sep 11 12:34:18.049299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.049151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="230ec79b-160b-4621-ae0c-17e368f30bb4" containerName="payload-processing"
Sep 11 12:34:18.049299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.049246    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9368df61-f662-4d3d-9ae4-0adaa5334eff" containerName="payload-pre-processing"
Sep 11 12:34:18.049299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.049262    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="230ec79b-160b-4621-ae0c-17e368f30bb4" containerName="payload-processing"
Sep 11 12:34:18.052854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.052833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.055324 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.055303    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-aaa824-openshift-default-dockercfg-bppqg\""
Sep 11 12:34:18.067550 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.067529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"]
Sep 11 12:34:18.090849 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.090810    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:18.145969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.145936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-credential-socket\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146112 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-token\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-podinfo\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146246 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-envoy\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-socket\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146339 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56bf75-c6aa-4c44-8a45-71c77c116220-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146410 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4j92\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-kube-api-access-x4j92\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-certs\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.146553 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.146539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-data\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247326 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-token\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-podinfo\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-envoy\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-socket\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56bf75-c6aa-4c44-8a45-71c77c116220-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4j92\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-kube-api-access-x4j92\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-certs\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-data\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-credential-socket\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.247982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.247939    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-socket\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.248238 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.248207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-credential-socket\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.248349 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.248246    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-data\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.248349 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.248333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-certs\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.248533 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.248507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56bf75-c6aa-4c44-8a45-71c77c116220-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.249972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.249950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-envoy\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.250157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.250140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-podinfo\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.255964 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.255936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-token\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.256152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.256130    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4j92\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-kube-api-access-x4j92\") pod \"e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") " pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.367426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.367370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:18.607063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.607020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:18.649787 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.649756    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"]
Sep 11 12:34:18.651736 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:34:18.651705    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c56bf75_c6aa_4c44_8a45_71c77c116220.slice/crio-1228f97f4e5b0e4a5efa8b7834e8d697c5d9c26be25b2c1efdebb10eb3ffd57f WatchSource:0}: Error finding container 1228f97f4e5b0e4a5efa8b7834e8d697c5d9c26be25b2c1efdebb10eb3ffd57f: Status 404 returned error can't find the container with id 1228f97f4e5b0e4a5efa8b7834e8d697c5d9c26be25b2c1efdebb10eb3ffd57f
Sep 11 12:34:18.653792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.653759    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:18.653897 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.653835    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:18.653897 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.653884    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:18.775747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.775632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" event={"ID":"3c56bf75-c6aa-4c44-8a45-71c77c116220","Type":"ContainerStarted","Data":"22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"}
Sep 11 12:34:18.775747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.775707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" event={"ID":"3c56bf75-c6aa-4c44-8a45-71c77c116220","Type":"ContainerStarted","Data":"1228f97f4e5b0e4a5efa8b7834e8d697c5d9c26be25b2c1efdebb10eb3ffd57f"}
Sep 11 12:34:18.797497 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.797445    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podStartSLOduration=0.797429026 podStartE2EDuration="797.429026ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:18.795550153 +0000 UTC m=+1194.186087497" watchObservedRunningTime="2026-09-11 12:34:18.797429026 +0000 UTC m=+1194.187966368"
Sep 11 12:34:18.942808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:18.942699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:19.367808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:19.367765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:19.373937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:19.373909    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:19.781341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:19.781252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:19.782591 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:19.782564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:21.008841 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.008787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:21.110913 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.110874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:21.684368 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.684330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:21.794480 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.794447    2561 generic.go:356] "Generic (PLEG): container finished" podID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerID="0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275" exitCode=0
Sep 11 12:34:21.794642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.794519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" event={"ID":"3548b9a1-9db3-4b02-a1ce-34064f2407fc","Type":"ContainerDied","Data":"0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"}
Sep 11 12:34:21.795014 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.794999    2561 scope.go:117] "RemoveContainer" containerID="0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"
Sep 11 12:34:21.796123 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.796094    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:21.796219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.796159    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:21.796219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:21.796194    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:22.000377 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.000289    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"]
Sep 11 12:34:22.004751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.004724    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.007734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.007701    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-aaa824-serving-cert\""
Sep 11 12:34:22.022409 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.022363    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"]
Sep 11 12:34:22.067185 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.067125    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:22.167731 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.167687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:22.187324 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.187294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.187513 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.187333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wbz\" (UniqueName: \"kubernetes.io/projected/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-kube-api-access-84wbz\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.187513 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.187359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.187513 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.187480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.267666 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.267590    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:22.288376 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.288349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.288489 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.288383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wbz\" (UniqueName: \"kubernetes.io/projected/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-kube-api-access-84wbz\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.288489 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.288437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.288648 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.288622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.289194 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.289173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.290762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.290747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.290817 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.290781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.295638 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.295615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wbz\" (UniqueName: \"kubernetes.io/projected/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-kube-api-access-84wbz\") pod \"maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.318156 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.318139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:22.523611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.523524    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:22.598706 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.598679    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"]
Sep 11 12:34:22.600805 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:34:22.600774    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89a193a_1f26_48d5_8d03_8c79a60c4d9d.slice/crio-a0b2de238610498cea435bc6bbed95791b375c7a04c6b0c60f222e01c59f8236 WatchSource:0}: Error finding container a0b2de238610498cea435bc6bbed95791b375c7a04c6b0c60f222e01c59f8236: Status 404 returned error can't find the container with id a0b2de238610498cea435bc6bbed95791b375c7a04c6b0c60f222e01c59f8236
Sep 11 12:34:22.601078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.601054    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:22.658488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.658425    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"]
Sep 11 12:34:22.663194 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.663175    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.665920 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.665898    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-aaa824\""
Sep 11 12:34:22.666996 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.666981    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-aaa824-dockercfg-sftcf\""
Sep 11 12:34:22.672337 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.672312    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"]
Sep 11 12:34:22.793823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.793738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt46k\" (UniqueName: \"kubernetes.io/projected/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-kube-api-access-gt46k\") pod \"payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.793823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.793820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.802633 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.802604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" event={"ID":"3548b9a1-9db3-4b02-a1ce-34064f2407fc","Type":"ContainerStarted","Data":"f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3"}
Sep 11 12:34:22.803976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.803942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" event={"ID":"d89a193a-1f26-48d5-8d03-8c79a60c4d9d","Type":"ContainerStarted","Data":"a0b2de238610498cea435bc6bbed95791b375c7a04c6b0c60f222e01c59f8236"}
Sep 11 12:34:22.895487 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.895449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt46k\" (UniqueName: \"kubernetes.io/projected/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-kube-api-access-gt46k\") pod \"payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.895659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.895519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.896197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.896175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.904022 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.904002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt46k\" (UniqueName: \"kubernetes.io/projected/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-kube-api-access-gt46k\") pod \"payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:22.977088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:22.977049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:23.231015 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.230980    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:34:23.231891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.231857    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 12:34:23.232057 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.231913    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 12:34:23.279112 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.279074    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"]
Sep 11 12:34:23.280155 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:34:23.280128    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b0f942_5aed_43ad_9eb1_a6a080e43e84.slice/crio-a9d3b93a2ae05200dfc0b81be133d83fe6669e05c2b820ac98c5a84cee929eea WatchSource:0}: Error finding container a9d3b93a2ae05200dfc0b81be133d83fe6669e05c2b820ac98c5a84cee929eea: Status 404 returned error can't find the container with id a9d3b93a2ae05200dfc0b81be133d83fe6669e05c2b820ac98c5a84cee929eea
Sep 11 12:34:23.811745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.811701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" event={"ID":"d89a193a-1f26-48d5-8d03-8c79a60c4d9d","Type":"ContainerStarted","Data":"0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc"}
Sep 11 12:34:23.811935 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.811887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:23.813533 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.813504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" event={"ID":"f9b0f942-5aed-43ad-9eb1-a6a080e43e84","Type":"ContainerStarted","Data":"525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae"}
Sep 11 12:34:23.813651 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.813562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" event={"ID":"f9b0f942-5aed-43ad-9eb1-a6a080e43e84","Type":"ContainerStarted","Data":"a9d3b93a2ae05200dfc0b81be133d83fe6669e05c2b820ac98c5a84cee929eea"}
Sep 11 12:34:23.813651 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.813643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:23.838150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.838097    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" podStartSLOduration=2.493461034 podStartE2EDuration="2.838078762s" totalImagesPullingTime="344.617728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:23.837523894 +0000 UTC m=+1199.228061237" watchObservedRunningTime="2026-09-11 12:34:23.838078762 +0000 UTC m=+1199.228616108"
Sep 11 12:34:23.863425 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:23.863346    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" podStartSLOduration=1.863322124 podStartE2EDuration="1.863322124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:23.859150725 +0000 UTC m=+1199.249688073" watchObservedRunningTime="2026-09-11 12:34:23.863322124 +0000 UTC m=+1199.253859467"
Sep 11 12:34:24.230351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:24.230318    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 12:34:24.230530 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:24.230371    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 12:34:25.231150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:25.231104    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 12:34:25.231579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:25.231168    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 12:34:25.622282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:25.622249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:34:26.230219 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:26.230185    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 12:34:26.230434 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:26.230238    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 12:34:27.230599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:27.230558    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 12:34:27.230978 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:27.230621    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 12:34:28.230884 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:28.230850    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 11 12:34:28.231247 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:28.230904    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 12:34:29.235230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:29.235200    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:34:29.236383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:29.236364    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:34:29.825224 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:29.825196    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:34:31.350713 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:31.350679    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:31.675085 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:31.674999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:31.988609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:31.988527    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:34.821529 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:34.821500    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:34:35.072263 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:35.072182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:35.170325 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:35.170285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:36.335018 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:36.334978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:36.437581 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:36.437541    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:36.602898 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:36.602819    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:36.785559 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:36.785514    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:36.887813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:36.887728    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:42.917464 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:42.917364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:45.775535 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:45.775490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:46.128883 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:46.128846    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:46.443457 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:46.443284    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:46.661732 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:46.661701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:47.953704 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:47.953661    2561 generic.go:356] "Generic (PLEG): container finished" podID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerID="22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874" exitCode=0
Sep 11 12:34:47.954119 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:47.953746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" event={"ID":"3c56bf75-c6aa-4c44-8a45-71c77c116220","Type":"ContainerDied","Data":"22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"}
Sep 11 12:34:47.954288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:47.954272    2561 scope.go:117] "RemoveContainer" containerID="22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"
Sep 11 12:34:47.955658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:47.955626    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:47.955747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:47.955732    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:47.955790 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:47.955777    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 12:34:48.965271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:48.965226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" event={"ID":"3c56bf75-c6aa-4c44-8a45-71c77c116220","Type":"ContainerStarted","Data":"c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27"}
Sep 11 12:34:49.164261 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:49.164218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:49.235245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:49.235159    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:49.367819 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:49.367781    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:49.369108 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:49.369077    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 12:34:49.369167 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:49.369135    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 12:34:49.781484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:49.781444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:50.324509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.324436    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"]
Sep 11 12:34:50.328976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.328951    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.331512 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.331484    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f4502d-serving-cert\""
Sep 11 12:34:50.348194 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.337940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"]
Sep 11 12:34:50.373795 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.371795    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 12:34:50.373795 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.371867    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 12:34:50.374572 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.374542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:50.473901 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.473855    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:50.479928 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.479901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.480066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.479947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3bd457-912c-44a1-83bf-d92a7c84e00a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.480066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.480035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxtk\" (UniqueName: \"kubernetes.io/projected/9d3bd457-912c-44a1-83bf-d92a7c84e00a-kube-api-access-5wxtk\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.480151 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.480069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.581447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.581343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.581447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.581417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3bd457-912c-44a1-83bf-d92a7c84e00a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.581690 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.581478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxtk\" (UniqueName: \"kubernetes.io/projected/9d3bd457-912c-44a1-83bf-d92a7c84e00a-kube-api-access-5wxtk\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.581690 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.581504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.581765 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:34:50.581694    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-serving-cert: secret "maas-api-e2e-shared-b-w3-f4502d-serving-cert" not found
Sep 11 12:34:50.581803 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:34:50.581778    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls podName:9d3bd457-912c-44a1-83bf-d92a7c84e00a nodeName:}" failed. No retries permitted until 2026-09-11 12:34:51.081757785 +0000 UTC m=+1226.472295104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" (UID: "9d3bd457-912c-44a1-83bf-d92a7c84e00a") : secret "maas-api-e2e-shared-b-w3-f4502d-serving-cert" not found
Sep 11 12:34:50.582169 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.582151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3bd457-912c-44a1-83bf-d92a7c84e00a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.583996 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.583965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.590026 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.590000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxtk\" (UniqueName: \"kubernetes.io/projected/9d3bd457-912c-44a1-83bf-d92a7c84e00a-kube-api-access-5wxtk\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:50.890374 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.890274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:50.990846 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:50.990814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:51.019987 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.019958    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"]
Sep 11 12:34:51.024003 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.023980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.026576 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.026501    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f4502d\""
Sep 11 12:34:51.034320 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.034295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"]
Sep 11 12:34:51.088223 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.088189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:51.088485 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:34:51.088326    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-serving-cert: secret "maas-api-e2e-shared-b-w3-f4502d-serving-cert" not found
Sep 11 12:34:51.088585 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:34:51.088533    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls podName:9d3bd457-912c-44a1-83bf-d92a7c84e00a nodeName:}" failed. No retries permitted until 2026-09-11 12:34:52.088516105 +0000 UTC m=+1227.479053425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" (UID: "9d3bd457-912c-44a1-83bf-d92a7c84e00a") : secret "maas-api-e2e-shared-b-w3-f4502d-serving-cert" not found
Sep 11 12:34:51.189690 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.189591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bq5\" (UniqueName: \"kubernetes.io/projected/b5fa7769-cf70-4d8d-b714-44a759196ba2-kube-api-access-t2bq5\") pod \"payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.189690 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.189671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fa7769-cf70-4d8d-b714-44a759196ba2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.290644 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.290612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bq5\" (UniqueName: \"kubernetes.io/projected/b5fa7769-cf70-4d8d-b714-44a759196ba2-kube-api-access-t2bq5\") pod \"payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.290805 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.290658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fa7769-cf70-4d8d-b714-44a759196ba2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.291279 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.291259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fa7769-cf70-4d8d-b714-44a759196ba2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.299785 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.299759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bq5\" (UniqueName: \"kubernetes.io/projected/b5fa7769-cf70-4d8d-b714-44a759196ba2-kube-api-access-t2bq5\") pod \"payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.337597 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.337568    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.371452 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.371420    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 12:34:51.371671 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.371642    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 12:34:51.634422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.634385    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"]
Sep 11 12:34:51.635891 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:34:51.635860    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fa7769_cf70_4d8d_b714_44a759196ba2.slice/crio-36219aa00af263964d7edb5bb1e65b5a707f5bcbe8ac1c947ce0e1a90940d2ad WatchSource:0}: Error finding container 36219aa00af263964d7edb5bb1e65b5a707f5bcbe8ac1c947ce0e1a90940d2ad: Status 404 returned error can't find the container with id 36219aa00af263964d7edb5bb1e65b5a707f5bcbe8ac1c947ce0e1a90940d2ad
Sep 11 12:34:51.675530 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.675499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:51.985545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.985443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp" event={"ID":"b5fa7769-cf70-4d8d-b714-44a759196ba2","Type":"ContainerStarted","Data":"37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244"}
Sep 11 12:34:51.985726 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.985669    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:34:51.985726 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:51.985705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp" event={"ID":"b5fa7769-cf70-4d8d-b714-44a759196ba2","Type":"ContainerStarted","Data":"36219aa00af263964d7edb5bb1e65b5a707f5bcbe8ac1c947ce0e1a90940d2ad"}
Sep 11 12:34:52.003049 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.002998    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp" podStartSLOduration=1.002966238 podStartE2EDuration="1.002966238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:52.001414538 +0000 UTC m=+1227.391951873" watchObservedRunningTime="2026-09-11 12:34:52.002966238 +0000 UTC m=+1227.393503617"
Sep 11 12:34:52.099433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.099362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:52.102418 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.102375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:52.173541 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.173505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:52.368581 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.368544    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 12:34:52.369054 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.368623    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 12:34:52.450105 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.450059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"]
Sep 11 12:34:52.994507 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.994469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" event={"ID":"9d3bd457-912c-44a1-83bf-d92a7c84e00a","Type":"ContainerStarted","Data":"c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d"}
Sep 11 12:34:52.994698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.994547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" event={"ID":"9d3bd457-912c-44a1-83bf-d92a7c84e00a","Type":"ContainerStarted","Data":"7b9f0bab0c225139cfb0d0a3d375fc5783dc218925da2c71ac477c4b94d727e6"}
Sep 11 12:34:52.994768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:52.994741    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:34:53.016241 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:53.016184    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" podStartSLOduration=2.662793392 podStartE2EDuration="3.016167348s" totalImagesPullingTime="353.373956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:53.013953897 +0000 UTC m=+1228.404491239" watchObservedRunningTime="2026-09-11 12:34:53.016167348 +0000 UTC m=+1228.406704689"
Sep 11 12:34:53.368589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:53.368546    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 11 12:34:53.368992 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:53.368620    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 11 12:34:54.372321 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:54.372290    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:54.373264 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:54.373246    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:34:57.307105 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:57.307070    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:34:59.007732 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:34:59.007701    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:35:00.113682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:00.113651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:02.996576 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:02.996542    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:35:11.554327 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:11.554293    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:13.595263 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:13.595232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:14.236441 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:14.236379    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:23.730813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:23.730696    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:35.004595 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:35.004552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:38.895792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:38.895758    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:49.193009 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:49.192975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:35:51.504708 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:35:51.504670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:03.521371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:03.521331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:11.337362 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:11.337320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:25.290381 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:25.290348    2561 scope.go:117] "RemoveContainer" containerID="a89cfe19bb58d2e4952c164388f98eaf33e7f92dc26ab7c0d92d06f6034aee75"
Sep 11 12:36:26.729645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:26.729604    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:26.799535 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:26.799498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:26.857787 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:26.857748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:26.952704 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:26.952672    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:27.011682 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:27.011587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:30.164581 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:30.164542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:35.239355 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:35.239316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:36:54.821762 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:36:54.821727    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:08.225900 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:08.225796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:12.116020 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:12.115896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:12.345982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:12.345948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:17.994854 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:17.994814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:18.048925 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:18.048886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:18.240856 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:18.240820    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:19.548413 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:19.548361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:19.632681 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:19.632646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:19.737812 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:19.737765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:20.448354 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:20.448321    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:20.545302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:20.545264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:20.645489 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:20.645452    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:21.883252 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:21.883217    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:27.878501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:27.878458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:27.935968 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:27.935919    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:28.526390 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:28.526348    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:34.883313 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:34.883274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:34.942610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:34.942576    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:35.041645 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:35.041611    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:35.146481 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:35.146409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:45.553620 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:45.553577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:45.791167 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:45.791128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:48.602145 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:48.602107    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:56.784115 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:56.784081    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:56.844903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:56.844864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:37:58.595367 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:37:58.595315    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:06.741983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:06.741944    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:12.693268 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:12.693229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:12.752332 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:12.752236    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:13.030260 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:13.030163    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:13.334102 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:13.334068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:13.940758 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:13.940724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:16.680414 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:16.680370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:16.780048 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:16.780002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:17.945296 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:17.945202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:18.004085 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:18.004053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:18.235807 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:18.235724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:18.461153 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:18.461121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:18.541885 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:18.541800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:23.556855 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:23.556815    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:27.990880 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:27.990838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:38.932958 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:38.932920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:41.552620 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:41.552580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:42.000687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:42.000647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:42.378855 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:42.378818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:42.584462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:42.584425    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:45.817284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:45.817246    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:45.916078 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:45.916040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:46.947089 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:46.947056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"]
Sep 11 12:38:46.951711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:46.951689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:46.954144 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:46.954121    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d18c79-serving-cert\""
Sep 11 12:38:46.962501 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:46.962476    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"]
Sep 11 12:38:46.981205 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:46.981153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:47.056476 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.056438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7sth\" (UniqueName: \"kubernetes.io/projected/1a2c5420-3b07-4c24-896f-05094bd13155-kube-api-access-h7sth\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.056667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.056486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2c5420-3b07-4c24-896f-05094bd13155-openshift-service-ca\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.056667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.056518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.056783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.056693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-metrics-certs\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.080245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.080205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:47.157484 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.157450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-metrics-certs\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.157693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.157565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7sth\" (UniqueName: \"kubernetes.io/projected/1a2c5420-3b07-4c24-896f-05094bd13155-kube-api-access-h7sth\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.157693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.157595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2c5420-3b07-4c24-896f-05094bd13155-openshift-service-ca\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.157693 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.157625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.157865 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:38:47.157771    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-serving-cert: secret "maas-api-e2e-api-b-d18c79-serving-cert" not found
Sep 11 12:38:47.157920 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:38:47.157870    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls podName:1a2c5420-3b07-4c24-896f-05094bd13155 nodeName:}" failed. No retries permitted until 2026-09-11 12:38:47.657848322 +0000 UTC m=+1463.048385642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls") pod "maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" (UID: "1a2c5420-3b07-4c24-896f-05094bd13155") : secret "maas-api-e2e-api-b-d18c79-serving-cert" not found
Sep 11 12:38:47.158237 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.158216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2c5420-3b07-4c24-896f-05094bd13155-openshift-service-ca\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.159932 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.159916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-metrics-certs\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.169941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.169911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7sth\" (UniqueName: \"kubernetes.io/projected/1a2c5420-3b07-4c24-896f-05094bd13155-kube-api-access-h7sth\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.491609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.491572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:47.591679 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.591637    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:47.656131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.656084    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"]
Sep 11 12:38:47.660671 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.660649    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.663549 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.663523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.665963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.665943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls\") pod \"maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.667871 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.667848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d18c79\""
Sep 11 12:38:47.667982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.667850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d18c79-dockercfg-d2wjq\""
Sep 11 12:38:47.676211 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.676187    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:47.676481 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.676461    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"]
Sep 11 12:38:47.764996 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.764917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b278ca31-662c-43f8-8779-b76ea6ee2f78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") " pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.765214 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.765079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x46h\" (UniqueName: \"kubernetes.io/projected/b278ca31-662c-43f8-8779-b76ea6ee2f78-kube-api-access-5x46h\") pod \"payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") " pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.864112 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.864081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:47.866570 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.866537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x46h\" (UniqueName: \"kubernetes.io/projected/b278ca31-662c-43f8-8779-b76ea6ee2f78-kube-api-access-5x46h\") pod \"payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") " pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.866715 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.866609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b278ca31-662c-43f8-8779-b76ea6ee2f78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") " pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.867325 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.867274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b278ca31-662c-43f8-8779-b76ea6ee2f78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") " pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.875966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.875938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x46h\" (UniqueName: \"kubernetes.io/projected/b278ca31-662c-43f8-8779-b76ea6ee2f78-kube-api-access-5x46h\") pod \"payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") " pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:47.983130 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:47.983095    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:48.146707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:48.146674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"]
Sep 11 12:38:48.148589 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:38:48.148562    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2c5420_3b07_4c24_896f_05094bd13155.slice/crio-9a967eb31aa6d0f8def20bc894e1b90c96b62e74fc9e36d4d24a570411911b06 WatchSource:0}: Error finding container 9a967eb31aa6d0f8def20bc894e1b90c96b62e74fc9e36d4d24a570411911b06: Status 404 returned error can't find the container with id 9a967eb31aa6d0f8def20bc894e1b90c96b62e74fc9e36d4d24a570411911b06
Sep 11 12:38:48.150082 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:48.150061    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:38:48.267558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:48.267515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" event={"ID":"1a2c5420-3b07-4c24-896f-05094bd13155","Type":"ContainerStarted","Data":"9a967eb31aa6d0f8def20bc894e1b90c96b62e74fc9e36d4d24a570411911b06"}
Sep 11 12:38:48.272628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:48.272604    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"]
Sep 11 12:38:48.274710 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:38:48.274687    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb278ca31_662c_43f8_8779_b76ea6ee2f78.slice/crio-19bda5f0c7a134d1a01bb413b751934972f96d8060216ce811146d61758a1dfd WatchSource:0}: Error finding container 19bda5f0c7a134d1a01bb413b751934972f96d8060216ce811146d61758a1dfd: Status 404 returned error can't find the container with id 19bda5f0c7a134d1a01bb413b751934972f96d8060216ce811146d61758a1dfd
Sep 11 12:38:49.273974 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.273940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" event={"ID":"1a2c5420-3b07-4c24-896f-05094bd13155","Type":"ContainerStarted","Data":"202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f"}
Sep 11 12:38:49.274415 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.274034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:38:49.275551 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.275526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4" event={"ID":"b278ca31-662c-43f8-8779-b76ea6ee2f78","Type":"ContainerStarted","Data":"f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7"}
Sep 11 12:38:49.275667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.275563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4" event={"ID":"b278ca31-662c-43f8-8779-b76ea6ee2f78","Type":"ContainerStarted","Data":"19bda5f0c7a134d1a01bb413b751934972f96d8060216ce811146d61758a1dfd"}
Sep 11 12:38:49.275667 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.275582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:38:49.309903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.309857    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" podStartSLOduration=2.931184064 podStartE2EDuration="3.309843823s" totalImagesPullingTime="378.659759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:49.308533833 +0000 UTC m=+1464.699071188" watchObservedRunningTime="2026-09-11 12:38:49.309843823 +0000 UTC m=+1464.700381215"
Sep 11 12:38:49.368474 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:49.368427    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4" podStartSLOduration=2.368412587 podStartE2EDuration="2.368412587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:49.365281792 +0000 UTC m=+1464.755819133" watchObservedRunningTime="2026-09-11 12:38:49.368412587 +0000 UTC m=+1464.758949924"
Sep 11 12:38:51.214509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:51.214471    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:38:55.287189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:38:55.287164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:39:00.283943 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:00.283911    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:39:07.909885 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:07.909844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:11.496867 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:11.496825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:14.000026 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:13.999986    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:19.626582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:19.626532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:23.964606 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:23.964573    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:28.263452 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:28.263383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:29.670415 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:29.670364    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:29.745952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:29.745917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:29.797349 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:29.797312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:29.860594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:29.860531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:29.918275 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:29.918232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:30.270164 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:30.270119    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:34.180390 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:34.180358    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:34.457163 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:34.457074    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:36.347542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:36.347499    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:36.448653 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:36.448616    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:36.586426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:36.586376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:37.019069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:37.019033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:39.263941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:39.263903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:43.615737 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:43.615701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:44.529301 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:44.529265    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:47.048610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:47.048574    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.107237 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.107201    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"]
Sep 11 12:39:50.109746 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.107440    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" podUID="80f14a47-9fdc-49c9-b47f-d79105195a32" containerName="maas-api" containerID="cri-o://a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba" gracePeriod=30
Sep 11 12:39:50.133754 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.133706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.167702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.167669    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"]
Sep 11 12:39:50.168158 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.168121    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk" podUID="4c6a4ec2-a307-48e9-82ce-c294421b1487" containerName="payload-processing" containerID="cri-o://54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5" gracePeriod=30
Sep 11 12:39:50.178540 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.178517    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"]
Sep 11 12:39:50.178795 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.178767    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw" podUID="5e14e285-5c91-43ed-ad46-e83b2e721bb4" containerName="payload-pre-processing" containerID="cri-o://756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591" gracePeriod=30
Sep 11 12:39:50.211171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.211110    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.260982 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.260957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.348809 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.348706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.404274 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.404242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.464727 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.464703    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:39:50.479046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.479020    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:39:50.517662 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.517628    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:39:50.596487 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-metrics-certs\") pod \"80f14a47-9fdc-49c9-b47f-d79105195a32\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") "
Sep 11 12:39:50.596659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxm5d\" (UniqueName: \"kubernetes.io/projected/80f14a47-9fdc-49c9-b47f-d79105195a32-kube-api-access-bxm5d\") pod \"80f14a47-9fdc-49c9-b47f-d79105195a32\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") "
Sep 11 12:39:50.596659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6a4ec2-a307-48e9-82ce-c294421b1487-plugins-config-volume\") pod \"4c6a4ec2-a307-48e9-82ce-c294421b1487\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") "
Sep 11 12:39:50.596659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls\") pod \"80f14a47-9fdc-49c9-b47f-d79105195a32\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") "
Sep 11 12:39:50.596800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66nb8\" (UniqueName: \"kubernetes.io/projected/4c6a4ec2-a307-48e9-82ce-c294421b1487-kube-api-access-66nb8\") pod \"4c6a4ec2-a307-48e9-82ce-c294421b1487\" (UID: \"4c6a4ec2-a307-48e9-82ce-c294421b1487\") "
Sep 11 12:39:50.596800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596714    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f14a47-9fdc-49c9-b47f-d79105195a32-openshift-service-ca\") pod \"80f14a47-9fdc-49c9-b47f-d79105195a32\" (UID: \"80f14a47-9fdc-49c9-b47f-d79105195a32\") "
Sep 11 12:39:50.596800 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e14e285-5c91-43ed-ad46-e83b2e721bb4-plugins-config-volume\") pod \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") "
Sep 11 12:39:50.596961 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.596828    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zmfs\" (UniqueName: \"kubernetes.io/projected/5e14e285-5c91-43ed-ad46-e83b2e721bb4-kube-api-access-9zmfs\") pod \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\" (UID: \"5e14e285-5c91-43ed-ad46-e83b2e721bb4\") "
Sep 11 12:39:50.597188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.597159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6a4ec2-a307-48e9-82ce-c294421b1487-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c6a4ec2-a307-48e9-82ce-c294421b1487" (UID: "4c6a4ec2-a307-48e9-82ce-c294421b1487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:50.597188 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.597158    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f14a47-9fdc-49c9-b47f-d79105195a32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80f14a47-9fdc-49c9-b47f-d79105195a32" (UID: "80f14a47-9fdc-49c9-b47f-d79105195a32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:50.597354 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.597220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e14e285-5c91-43ed-ad46-e83b2e721bb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e14e285-5c91-43ed-ad46-e83b2e721bb4" (UID: "5e14e285-5c91-43ed-ad46-e83b2e721bb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:50.598889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.598841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80f14a47-9fdc-49c9-b47f-d79105195a32" (UID: "80f14a47-9fdc-49c9-b47f-d79105195a32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:50.599046 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.599021    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80f14a47-9fdc-49c9-b47f-d79105195a32" (UID: "80f14a47-9fdc-49c9-b47f-d79105195a32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:50.599178 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.599157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f14a47-9fdc-49c9-b47f-d79105195a32-kube-api-access-bxm5d" (OuterVolumeSpecName: "kube-api-access-bxm5d") pod "80f14a47-9fdc-49c9-b47f-d79105195a32" (UID: "80f14a47-9fdc-49c9-b47f-d79105195a32"). InnerVolumeSpecName "kube-api-access-bxm5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:50.599236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.599185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6a4ec2-a307-48e9-82ce-c294421b1487-kube-api-access-66nb8" (OuterVolumeSpecName: "kube-api-access-66nb8") pod "4c6a4ec2-a307-48e9-82ce-c294421b1487" (UID: "4c6a4ec2-a307-48e9-82ce-c294421b1487"). InnerVolumeSpecName "kube-api-access-66nb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:50.599461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.599438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e14e285-5c91-43ed-ad46-e83b2e721bb4-kube-api-access-9zmfs" (OuterVolumeSpecName: "kube-api-access-9zmfs") pod "5e14e285-5c91-43ed-ad46-e83b2e721bb4" (UID: "5e14e285-5c91-43ed-ad46-e83b2e721bb4"). InnerVolumeSpecName "kube-api-access-9zmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:50.615595 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.615566    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e14e285-5c91-43ed-ad46-e83b2e721bb4" containerID="756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591" exitCode=0
Sep 11 12:39:50.615688 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.615618    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"
Sep 11 12:39:50.615688 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.615643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw" event={"ID":"5e14e285-5c91-43ed-ad46-e83b2e721bb4","Type":"ContainerDied","Data":"756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591"}
Sep 11 12:39:50.615768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.615695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw" event={"ID":"5e14e285-5c91-43ed-ad46-e83b2e721bb4","Type":"ContainerDied","Data":"1bebb6d8ed68a29704374bdd8bc81f7b38df032044196586c75480dc4dd0b1e4"}
Sep 11 12:39:50.615768 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.615720    2561 scope.go:117] "RemoveContainer" containerID="756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591"
Sep 11 12:39:50.617714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.617640    2561 generic.go:356] "Generic (PLEG): container finished" podID="4c6a4ec2-a307-48e9-82ce-c294421b1487" containerID="54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5" exitCode=0
Sep 11 12:39:50.617714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.617687    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"
Sep 11 12:39:50.617895 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.617706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk" event={"ID":"4c6a4ec2-a307-48e9-82ce-c294421b1487","Type":"ContainerDied","Data":"54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5"}
Sep 11 12:39:50.617895 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.617770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk" event={"ID":"4c6a4ec2-a307-48e9-82ce-c294421b1487","Type":"ContainerDied","Data":"1c707dd2b26c92b9010647e1ed2102b3e093cb8695553350333ce70f3ebd77d8"}
Sep 11 12:39:50.619354 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.619329    2561 generic.go:356] "Generic (PLEG): container finished" podID="80f14a47-9fdc-49c9-b47f-d79105195a32" containerID="a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba" exitCode=0
Sep 11 12:39:50.619493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.619428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" event={"ID":"80f14a47-9fdc-49c9-b47f-d79105195a32","Type":"ContainerDied","Data":"a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba"}
Sep 11 12:39:50.619493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.619465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58" event={"ID":"80f14a47-9fdc-49c9-b47f-d79105195a32","Type":"ContainerDied","Data":"913da58caf2e854aed1eed5a0754be59dd326abe0bfc284408330846f142576c"}
Sep 11 12:39:50.619493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.619486    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"
Sep 11 12:39:50.625392 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.625372    2561 scope.go:117] "RemoveContainer" containerID="756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591"
Sep 11 12:39:50.625749 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:39:50.625709    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591\": container with ID starting with 756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591 not found: ID does not exist" containerID="756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591"
Sep 11 12:39:50.625842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.625752    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591"} err="failed to get container status \"756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591\": rpc error: code = NotFound desc = could not find container \"756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591\": container with ID starting with 756207ef25da5677d699dfadccaef9fa2a03cece42c201b7ed8df01773fb8591 not found: ID does not exist"
Sep 11 12:39:50.625842 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.625775    2561 scope.go:117] "RemoveContainer" containerID="54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5"
Sep 11 12:39:50.635002 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.634960    2561 scope.go:117] "RemoveContainer" containerID="54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5"
Sep 11 12:39:50.635220 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:39:50.635202    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5\": container with ID starting with 54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5 not found: ID does not exist" containerID="54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5"
Sep 11 12:39:50.635277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.635225    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5"} err="failed to get container status \"54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5\": rpc error: code = NotFound desc = could not find container \"54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5\": container with ID starting with 54fa23df063f5704f2bbb84bdc1c1f0627d1a6069ad20c663af980261586b6f5 not found: ID does not exist"
Sep 11 12:39:50.635277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.635242    2561 scope.go:117] "RemoveContainer" containerID="a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba"
Sep 11 12:39:50.640050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.640026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"]
Sep 11 12:39:50.642855 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.642831    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1a8ee3-75bdddf99d-8m7pw"]
Sep 11 12:39:50.643978 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.643961    2561 scope.go:117] "RemoveContainer" containerID="a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba"
Sep 11 12:39:50.644247 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:39:50.644231    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba\": container with ID starting with a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba not found: ID does not exist" containerID="a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba"
Sep 11 12:39:50.644309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.644253    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba"} err="failed to get container status \"a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba\": rpc error: code = NotFound desc = could not find container \"a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba\": container with ID starting with a74dc57e7f47d3fede7455f2f07b913b52c44616376cdf20d78b974e6c5ee9ba not found: ID does not exist"
Sep 11 12:39:50.652689 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.652667    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"]
Sep 11 12:39:50.659919 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.659900    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a8ee3-665c9474b7-5np58"]
Sep 11 12:39:50.670247 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.670226    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"]
Sep 11 12:39:50.673725 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.673705    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a8ee3-574bb8d64f-xjxdk"]
Sep 11 12:39:50.692226 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.692202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:50.698502 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698483    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66nb8\" (UniqueName: \"kubernetes.io/projected/4c6a4ec2-a307-48e9-82ce-c294421b1487-kube-api-access-66nb8\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698505    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f14a47-9fdc-49c9-b47f-d79105195a32-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698514    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e14e285-5c91-43ed-ad46-e83b2e721bb4-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698523    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zmfs\" (UniqueName: \"kubernetes.io/projected/5e14e285-5c91-43ed-ad46-e83b2e721bb4-kube-api-access-9zmfs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698531    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698541    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxm5d\" (UniqueName: \"kubernetes.io/projected/80f14a47-9fdc-49c9-b47f-d79105195a32-kube-api-access-bxm5d\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698549    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6a4ec2-a307-48e9-82ce-c294421b1487-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:50.698568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:50.698556    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f14a47-9fdc-49c9-b47f-d79105195a32-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:51.107623 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:51.107591    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c6a4ec2-a307-48e9-82ce-c294421b1487" path="/var/lib/kubelet/pods/4c6a4ec2-a307-48e9-82ce-c294421b1487/volumes"
Sep 11 12:39:51.108006 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:51.107992    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e14e285-5c91-43ed-ad46-e83b2e721bb4" path="/var/lib/kubelet/pods/5e14e285-5c91-43ed-ad46-e83b2e721bb4/volumes"
Sep 11 12:39:51.108379 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:51.108367    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f14a47-9fdc-49c9-b47f-d79105195a32" path="/var/lib/kubelet/pods/80f14a47-9fdc-49c9-b47f-d79105195a32/volumes"
Sep 11 12:39:52.617485 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.617446    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"]
Sep 11 12:39:52.618312 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.618273    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" podUID="1a2c5420-3b07-4c24-896f-05094bd13155" containerName="maas-api" containerID="cri-o://202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f" gracePeriod=30
Sep 11 12:39:52.639480 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.639438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:52.669478 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.669436    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"]
Sep 11 12:39:52.669714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.669686    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4" podUID="b278ca31-662c-43f8-8779-b76ea6ee2f78" containerName="payload-processing" containerID="cri-o://f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7" gracePeriod=30
Sep 11 12:39:52.737203 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.737168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:52.834779 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.834720    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:52.891985 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.891949    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:52.933029 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.932695    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:39:52.968827 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:52.968797    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:39:53.024015 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.023986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7sth\" (UniqueName: \"kubernetes.io/projected/1a2c5420-3b07-4c24-896f-05094bd13155-kube-api-access-h7sth\") pod \"1a2c5420-3b07-4c24-896f-05094bd13155\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") "
Sep 11 12:39:53.024186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.024078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls\") pod \"1a2c5420-3b07-4c24-896f-05094bd13155\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") "
Sep 11 12:39:53.024186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.024108    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-metrics-certs\") pod \"1a2c5420-3b07-4c24-896f-05094bd13155\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") "
Sep 11 12:39:53.024186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.024145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2c5420-3b07-4c24-896f-05094bd13155-openshift-service-ca\") pod \"1a2c5420-3b07-4c24-896f-05094bd13155\" (UID: \"1a2c5420-3b07-4c24-896f-05094bd13155\") "
Sep 11 12:39:53.024568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.024547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2c5420-3b07-4c24-896f-05094bd13155-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a2c5420-3b07-4c24-896f-05094bd13155" (UID: "1a2c5420-3b07-4c24-896f-05094bd13155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:53.026155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.026128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a2c5420-3b07-4c24-896f-05094bd13155" (UID: "1a2c5420-3b07-4c24-896f-05094bd13155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:53.026155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.026141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2c5420-3b07-4c24-896f-05094bd13155-kube-api-access-h7sth" (OuterVolumeSpecName: "kube-api-access-h7sth") pod "1a2c5420-3b07-4c24-896f-05094bd13155" (UID: "1a2c5420-3b07-4c24-896f-05094bd13155"). InnerVolumeSpecName "kube-api-access-h7sth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:53.026155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.026146    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a2c5420-3b07-4c24-896f-05094bd13155" (UID: "1a2c5420-3b07-4c24-896f-05094bd13155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:53.125663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.125638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b278ca31-662c-43f8-8779-b76ea6ee2f78-plugins-config-volume\") pod \"b278ca31-662c-43f8-8779-b76ea6ee2f78\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") "
Sep 11 12:39:53.125825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.125708    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x46h\" (UniqueName: \"kubernetes.io/projected/b278ca31-662c-43f8-8779-b76ea6ee2f78-kube-api-access-5x46h\") pod \"b278ca31-662c-43f8-8779-b76ea6ee2f78\" (UID: \"b278ca31-662c-43f8-8779-b76ea6ee2f78\") "
Sep 11 12:39:53.126003 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.125991    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7sth\" (UniqueName: \"kubernetes.io/projected/1a2c5420-3b07-4c24-896f-05094bd13155-kube-api-access-h7sth\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.126047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.125995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b278ca31-662c-43f8-8779-b76ea6ee2f78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b278ca31-662c-43f8-8779-b76ea6ee2f78" (UID: "b278ca31-662c-43f8-8779-b76ea6ee2f78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:53.126047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.126005    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.126047 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.126042    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2c5420-3b07-4c24-896f-05094bd13155-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.126142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.126052    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a2c5420-3b07-4c24-896f-05094bd13155-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.127694 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.127674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b278ca31-662c-43f8-8779-b76ea6ee2f78-kube-api-access-5x46h" (OuterVolumeSpecName: "kube-api-access-5x46h") pod "b278ca31-662c-43f8-8779-b76ea6ee2f78" (UID: "b278ca31-662c-43f8-8779-b76ea6ee2f78"). InnerVolumeSpecName "kube-api-access-5x46h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:53.156743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.156667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:53.226805 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.226774    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x46h\" (UniqueName: \"kubernetes.io/projected/b278ca31-662c-43f8-8779-b76ea6ee2f78-kube-api-access-5x46h\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.226805 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.226801    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b278ca31-662c-43f8-8779-b76ea6ee2f78-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.644284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.644246    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a2c5420-3b07-4c24-896f-05094bd13155" containerID="202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f" exitCode=0
Sep 11 12:39:53.644701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.644345    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"
Sep 11 12:39:53.644701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.644380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" event={"ID":"1a2c5420-3b07-4c24-896f-05094bd13155","Type":"ContainerDied","Data":"202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f"}
Sep 11 12:39:53.644701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.644451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b" event={"ID":"1a2c5420-3b07-4c24-896f-05094bd13155","Type":"ContainerDied","Data":"9a967eb31aa6d0f8def20bc894e1b90c96b62e74fc9e36d4d24a570411911b06"}
Sep 11 12:39:53.644701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.644481    2561 scope.go:117] "RemoveContainer" containerID="202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f"
Sep 11 12:39:53.646206 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.646185    2561 generic.go:356] "Generic (PLEG): container finished" podID="b278ca31-662c-43f8-8779-b76ea6ee2f78" containerID="f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7" exitCode=0
Sep 11 12:39:53.646325 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.646240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4" event={"ID":"b278ca31-662c-43f8-8779-b76ea6ee2f78","Type":"ContainerDied","Data":"f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7"}
Sep 11 12:39:53.646325 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.646249    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"
Sep 11 12:39:53.646325 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.646283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4" event={"ID":"b278ca31-662c-43f8-8779-b76ea6ee2f78","Type":"ContainerDied","Data":"19bda5f0c7a134d1a01bb413b751934972f96d8060216ce811146d61758a1dfd"}
Sep 11 12:39:53.655869 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.655848    2561 scope.go:117] "RemoveContainer" containerID="202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f"
Sep 11 12:39:53.656156 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:39:53.656133    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f\": container with ID starting with 202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f not found: ID does not exist" containerID="202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f"
Sep 11 12:39:53.656351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.656165    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f"} err="failed to get container status \"202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f\": rpc error: code = NotFound desc = could not find container \"202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f\": container with ID starting with 202638327f7328f383eea5739039546a6475cc3a09e17b2705848299485eb85f not found: ID does not exist"
Sep 11 12:39:53.656351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.656186    2561 scope.go:117] "RemoveContainer" containerID="f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7"
Sep 11 12:39:53.660307 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.660285    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"]
Sep 11 12:39:53.665274 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.665259    2561 scope.go:117] "RemoveContainer" containerID="f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7"
Sep 11 12:39:53.665574 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:39:53.665557    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7\": container with ID starting with f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7 not found: ID does not exist" containerID="f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7"
Sep 11 12:39:53.665622 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.665580    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7"} err="failed to get container status \"f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7\": rpc error: code = NotFound desc = could not find container \"f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7\": container with ID starting with f86ed11eac6a1f96353ff614c3becb07da0245a40989ffbf07cf8b47dfdadbf7 not found: ID does not exist"
Sep 11 12:39:53.665937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.665920    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d18c79-845bfdfff-4vp8b"]
Sep 11 12:39:53.678517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.678431    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"]
Sep 11 12:39:53.680542 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:53.680523    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d18c79-68468448fc-4vnw4"]
Sep 11 12:39:55.107572 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:55.107543    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a2c5420-3b07-4c24-896f-05094bd13155" path="/var/lib/kubelet/pods/1a2c5420-3b07-4c24-896f-05094bd13155/volumes"
Sep 11 12:39:55.107958 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:55.107944    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b278ca31-662c-43f8-8779-b76ea6ee2f78" path="/var/lib/kubelet/pods/b278ca31-662c-43f8-8779-b76ea6ee2f78/volumes"
Sep 11 12:39:56.621382 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:56.621342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:56.687609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:56.687560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:56.846151 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:56.846115    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:39:58.986764 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:39:58.986723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:00.219056 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:00.219020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.121126 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.121083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"]
Sep 11 12:40:10.121652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.121347    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" podUID="be3b6a74-ae75-403e-8734-bc7922a704f6" containerName="maas-api" containerID="cri-o://f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7" gracePeriod=30
Sep 11 12:40:10.143843 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.143804    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.179596 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.179564    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"]
Sep 11 12:40:10.179830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.179784    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6" podUID="4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" containerName="payload-pre-processing" containerID="cri-o://159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d" gracePeriod=30
Sep 11 12:40:10.207922 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.207890    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.261882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.261844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.340256 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.340224    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.400973 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.400941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.433037 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.433012    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:40:10.469731 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.469698    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:40:10.489318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.488800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-metrics-certs\") pod \"be3b6a74-ae75-403e-8734-bc7922a704f6\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") "
Sep 11 12:40:10.489318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.488898    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-plugins-config-volume\") pod \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") "
Sep 11 12:40:10.489318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.488932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b6a74-ae75-403e-8734-bc7922a704f6-openshift-service-ca\") pod \"be3b6a74-ae75-403e-8734-bc7922a704f6\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") "
Sep 11 12:40:10.489318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.488969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szxdz\" (UniqueName: \"kubernetes.io/projected/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-kube-api-access-szxdz\") pod \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\" (UID: \"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637\") "
Sep 11 12:40:10.489318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.489053    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls\") pod \"be3b6a74-ae75-403e-8734-bc7922a704f6\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") "
Sep 11 12:40:10.489318 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.489109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkfrj\" (UniqueName: \"kubernetes.io/projected/be3b6a74-ae75-403e-8734-bc7922a704f6-kube-api-access-vkfrj\") pod \"be3b6a74-ae75-403e-8734-bc7922a704f6\" (UID: \"be3b6a74-ae75-403e-8734-bc7922a704f6\") "
Sep 11 12:40:10.489821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.489527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3b6a74-ae75-403e-8734-bc7922a704f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be3b6a74-ae75-403e-8734-bc7922a704f6" (UID: "be3b6a74-ae75-403e-8734-bc7922a704f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:10.489933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.489909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" (UID: "4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:10.492264 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.492235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be3b6a74-ae75-403e-8734-bc7922a704f6" (UID: "be3b6a74-ae75-403e-8734-bc7922a704f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:40:10.492462 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.492429    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be3b6a74-ae75-403e-8734-bc7922a704f6" (UID: "be3b6a74-ae75-403e-8734-bc7922a704f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:40:10.492663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.492644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-kube-api-access-szxdz" (OuterVolumeSpecName: "kube-api-access-szxdz") pod "4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" (UID: "4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637"). InnerVolumeSpecName "kube-api-access-szxdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:10.492906 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.492887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3b6a74-ae75-403e-8734-bc7922a704f6-kube-api-access-vkfrj" (OuterVolumeSpecName: "kube-api-access-vkfrj") pod "be3b6a74-ae75-403e-8734-bc7922a704f6" (UID: "be3b6a74-ae75-403e-8734-bc7922a704f6"). InnerVolumeSpecName "kube-api-access-vkfrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:10.590148 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.590110    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxdz\" (UniqueName: \"kubernetes.io/projected/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-kube-api-access-szxdz\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.590148 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.590144    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.590148 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.590155    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkfrj\" (UniqueName: \"kubernetes.io/projected/be3b6a74-ae75-403e-8734-bc7922a704f6-kube-api-access-vkfrj\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.590385 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.590166    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3b6a74-ae75-403e-8734-bc7922a704f6-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.590385 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.590175    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.590385 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.590185    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b6a74-ae75-403e-8734-bc7922a704f6-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.703052 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.702957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:10.749707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.748485    2561 generic.go:356] "Generic (PLEG): container finished" podID="4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" containerID="159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d" exitCode=0
Sep 11 12:40:10.749707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.748550    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"
Sep 11 12:40:10.749707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.748604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6" event={"ID":"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637","Type":"ContainerDied","Data":"159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d"}
Sep 11 12:40:10.749707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.748658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6" event={"ID":"4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637","Type":"ContainerDied","Data":"84d0fe3f02d0af8eca21347a48228d5300c2cc10d46f30bf76e53dcaf7f58f9e"}
Sep 11 12:40:10.749707 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.748692    2561 scope.go:117] "RemoveContainer" containerID="159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d"
Sep 11 12:40:10.753798 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.753764    2561 generic.go:356] "Generic (PLEG): container finished" podID="be3b6a74-ae75-403e-8734-bc7922a704f6" containerID="f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7" exitCode=0
Sep 11 12:40:10.753966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.753945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" event={"ID":"be3b6a74-ae75-403e-8734-bc7922a704f6","Type":"ContainerDied","Data":"f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7"}
Sep 11 12:40:10.754060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.753996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7" event={"ID":"be3b6a74-ae75-403e-8734-bc7922a704f6","Type":"ContainerDied","Data":"666624d6cc8a1aa35df061373c3d8e2c76edfe230a98f05408b5024e9f738342"}
Sep 11 12:40:10.754129 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.754069    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"
Sep 11 12:40:10.771938 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.771820    2561 scope.go:117] "RemoveContainer" containerID="159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d"
Sep 11 12:40:10.772776 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:40:10.772747    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d\": container with ID starting with 159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d not found: ID does not exist" containerID="159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d"
Sep 11 12:40:10.772882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.772789    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d"} err="failed to get container status \"159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d\": rpc error: code = NotFound desc = could not find container \"159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d\": container with ID starting with 159a9b58cd0924d3d91ebce8c88082ad7084d2bb48536a0bd4a383cbb322621d not found: ID does not exist"
Sep 11 12:40:10.772882 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.772815    2561 scope.go:117] "RemoveContainer" containerID="f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7"
Sep 11 12:40:10.788182 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.788089    2561 scope.go:117] "RemoveContainer" containerID="f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7"
Sep 11 12:40:10.788672 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:40:10.788617    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7\": container with ID starting with f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7 not found: ID does not exist" containerID="f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7"
Sep 11 12:40:10.788672 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.788655    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7"} err="failed to get container status \"f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7\": rpc error: code = NotFound desc = could not find container \"f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7\": container with ID starting with f822ad22a7028955906e637d1315cb8380fc9993ee9b7adc25ab22fc6b5af5c7 not found: ID does not exist"
Sep 11 12:40:10.789974 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.789949    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"]
Sep 11 12:40:10.793549 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.793528    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-44f372-97b4475d6-mwjl6"]
Sep 11 12:40:10.801456 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.801314    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"]
Sep 11 12:40:10.807695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:10.807609    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-44f372-78cc8f4d5f-dkww7"]
Sep 11 12:40:11.107366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:11.107328    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" path="/var/lib/kubelet/pods/4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637/volumes"
Sep 11 12:40:11.107810 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:11.107793    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be3b6a74-ae75-403e-8734-bc7922a704f6" path="/var/lib/kubelet/pods/be3b6a74-ae75-403e-8734-bc7922a704f6/volumes"
Sep 11 12:40:12.316831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:12.316794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:16.188042 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:16.188001    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:16.288937 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:16.288896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:17.125993 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:17.125958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:17.346976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:17.346943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:17.447705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:17.447620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:17.599181 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:17.599145    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:18.013431 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:18.013385    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:22.788561 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:22.788523    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:24.444712 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:24.444677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:28.142896 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.142855    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:28.703373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.703332    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"]
Sep 11 12:40:28.703650 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.703612    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" podUID="35380b8a-2c03-4cfc-9e92-e5ea9da0f126" containerName="maas-api" containerID="cri-o://21228f43449f3e0377b0ae18ae2c3186443b5b63ff332d43a99c778aef600347" gracePeriod=30
Sep 11 12:40:28.729069 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.729035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:28.794575 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.794541    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:28.848981 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.848941    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:28.857004 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.856955    2561 generic.go:356] "Generic (PLEG): container finished" podID="35380b8a-2c03-4cfc-9e92-e5ea9da0f126" containerID="21228f43449f3e0377b0ae18ae2c3186443b5b63ff332d43a99c778aef600347" exitCode=0
Sep 11 12:40:28.857143 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.857034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" event={"ID":"35380b8a-2c03-4cfc-9e92-e5ea9da0f126","Type":"ContainerDied","Data":"21228f43449f3e0377b0ae18ae2c3186443b5b63ff332d43a99c778aef600347"}
Sep 11 12:40:28.915539 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.915493    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:28.959659 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.959591    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:40:28.969522 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:28.969488    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:29.076878 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.076839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-metrics-certs\") pod \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") "
Sep 11 12:40:29.077053 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.076912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dk55\" (UniqueName: \"kubernetes.io/projected/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-kube-api-access-8dk55\") pod \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") "
Sep 11 12:40:29.077053 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.076982    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-openshift-service-ca\") pod \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") "
Sep 11 12:40:29.077169 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.077051    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls\") pod \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\" (UID: \"35380b8a-2c03-4cfc-9e92-e5ea9da0f126\") "
Sep 11 12:40:29.077600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.077557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35380b8a-2c03-4cfc-9e92-e5ea9da0f126" (UID: "35380b8a-2c03-4cfc-9e92-e5ea9da0f126"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:29.079307 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.079275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35380b8a-2c03-4cfc-9e92-e5ea9da0f126" (UID: "35380b8a-2c03-4cfc-9e92-e5ea9da0f126"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:40:29.079629 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.079598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-kube-api-access-8dk55" (OuterVolumeSpecName: "kube-api-access-8dk55") pod "35380b8a-2c03-4cfc-9e92-e5ea9da0f126" (UID: "35380b8a-2c03-4cfc-9e92-e5ea9da0f126"). InnerVolumeSpecName "kube-api-access-8dk55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:29.079745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.079689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "35380b8a-2c03-4cfc-9e92-e5ea9da0f126" (UID: "35380b8a-2c03-4cfc-9e92-e5ea9da0f126"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:40:29.178618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.178583    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.178618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.178613    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dk55\" (UniqueName: \"kubernetes.io/projected/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-kube-api-access-8dk55\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.178618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.178624    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.179050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.178633    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35380b8a-2c03-4cfc-9e92-e5ea9da0f126-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.335351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.335318    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:29.863585 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.863551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn" event={"ID":"35380b8a-2c03-4cfc-9e92-e5ea9da0f126","Type":"ContainerDied","Data":"71b24da0d45dadbc18e0fc7932454acb69345a0d3cc3f4fa7335a7dc572aed2d"}
Sep 11 12:40:29.863764 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.863602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"
Sep 11 12:40:29.863764 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.863613    2561 scope.go:117] "RemoveContainer" containerID="21228f43449f3e0377b0ae18ae2c3186443b5b63ff332d43a99c778aef600347"
Sep 11 12:40:29.880651 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.880623    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"]
Sep 11 12:40:29.884162 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:29.884140    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a50cf9-678f59c8d4-t6vkn"]
Sep 11 12:40:31.107057 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:31.107022    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35380b8a-2c03-4cfc-9e92-e5ea9da0f126" path="/var/lib/kubelet/pods/35380b8a-2c03-4cfc-9e92-e5ea9da0f126/volumes"
Sep 11 12:40:34.973373 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:34.973338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:35.028865 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:35.028828    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:35.216248 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:35.216215    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"]
Sep 11 12:40:35.216509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:35.216481    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy" containerID="cri-o://f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3" gracePeriod=30
Sep 11 12:40:35.236736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:35.236649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:40.479847 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.479790    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:40:40.488942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.488920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-socket\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.488950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-envoy\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.488977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-data\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-kube-api-access-7lbqw\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-certs\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489283 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-credential-socket\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489283 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489142    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-token\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489283 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istiod-ca-cert\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489283 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489184    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:40:40.489283 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489206    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-podinfo\") pod \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\" (UID: \"3548b9a1-9db3-4b02-a1ce-34064f2407fc\") "
Sep 11 12:40:40.489565 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489290    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:40:40.489565 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:40:40.489565 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489394    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:40:40.489745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489653    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.489745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489675    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-credential-socket\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.489745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489689    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-workload-socket\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.489745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:40.489745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.489703    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-data\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.491123 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.491101    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:40:40.491426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.491387    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 12:40:40.491610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.491586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-kube-api-access-7lbqw" (OuterVolumeSpecName: "kube-api-access-7lbqw") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "kube-api-access-7lbqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:40.491652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.491598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "3548b9a1-9db3-4b02-a1ce-34064f2407fc" (UID: "3548b9a1-9db3-4b02-a1ce-34064f2407fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:40.590850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.590811    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-token\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.590850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.590843    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istiod-ca-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.590850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.590852    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-podinfo\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.590850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.590860    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3548b9a1-9db3-4b02-a1ce-34064f2407fc-istio-envoy\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.591113 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.590870    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lbqw\" (UniqueName: \"kubernetes.io/projected/3548b9a1-9db3-4b02-a1ce-34064f2407fc-kube-api-access-7lbqw\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:40.919634 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.919599    2561 generic.go:356] "Generic (PLEG): container finished" podID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerID="f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3" exitCode=0
Sep 11 12:40:40.919816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.919649    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"
Sep 11 12:40:40.919816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.919686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" event={"ID":"3548b9a1-9db3-4b02-a1ce-34064f2407fc","Type":"ContainerDied","Data":"f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3"}
Sep 11 12:40:40.919816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.919741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg" event={"ID":"3548b9a1-9db3-4b02-a1ce-34064f2407fc","Type":"ContainerDied","Data":"6c24a0d9b5ed43afbba605d54be04287b4f204113929486dc4df161362446873"}
Sep 11 12:40:40.919816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.919766    2561 scope.go:117] "RemoveContainer" containerID="f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3"
Sep 11 12:40:40.929001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.928982    2561 scope.go:117] "RemoveContainer" containerID="0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"
Sep 11 12:40:40.937569 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.937551    2561 scope.go:117] "RemoveContainer" containerID="f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3"
Sep 11 12:40:40.937834 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:40:40.937814    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3\": container with ID starting with f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3 not found: ID does not exist" containerID="f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3"
Sep 11 12:40:40.937895 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.937842    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3"} err="failed to get container status \"f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3\": rpc error: code = NotFound desc = could not find container \"f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3\": container with ID starting with f63762ff7b69b390727ea747b26fc00adf9f59427ad8972a2702d489473ac8f3 not found: ID does not exist"
Sep 11 12:40:40.937895 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.937859    2561 scope.go:117] "RemoveContainer" containerID="0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"
Sep 11 12:40:40.938099 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:40:40.938084    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275\": container with ID starting with 0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275 not found: ID does not exist" containerID="0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"
Sep 11 12:40:40.938142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.938102    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275"} err="failed to get container status \"0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275\": rpc error: code = NotFound desc = could not find container \"0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275\": container with ID starting with 0190aac5ac38193feb48675fe678eeebcb1387ed6c83ac58326015eac99b7275 not found: ID does not exist"
Sep 11 12:40:40.940959 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.940936    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"]
Sep 11 12:40:40.945060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:40.945041    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a50cf9-openshift-default-79cdbc5464-wcrcg"]
Sep 11 12:40:41.107221 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:41.107187    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" path="/var/lib/kubelet/pods/3548b9a1-9db3-4b02-a1ce-34064f2407fc/volumes"
Sep 11 12:40:42.127751 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:42.127702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:42.489091 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:42.489004    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:47.206098 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:47.206060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:57.971091 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:57.971057    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"]
Sep 11 12:40:57.971556 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:57.971256    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" podUID="70f2e1ae-a69a-4f05-8987-8a66c82acf60" containerName="maas-api" containerID="cri-o://1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea" gracePeriod=30
Sep 11 12:40:57.994828 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:57.994781    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:58.070071 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.070038    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:58.127088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.127058    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:58.226739 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.226677    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:40:58.233531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.233500    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:58.313060 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.313025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:58.356386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.356355    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f2e1ae-a69a-4f05-8987-8a66c82acf60-openshift-service-ca\") pod \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") "
Sep 11 12:40:58.356587 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.356419    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dktcg\" (UniqueName: \"kubernetes.io/projected/70f2e1ae-a69a-4f05-8987-8a66c82acf60-kube-api-access-dktcg\") pod \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") "
Sep 11 12:40:58.356587 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.356450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-metrics-certs\") pod \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") "
Sep 11 12:40:58.356587 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.356520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls\") pod \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\" (UID: \"70f2e1ae-a69a-4f05-8987-8a66c82acf60\") "
Sep 11 12:40:58.356801 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.356777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f2e1ae-a69a-4f05-8987-8a66c82acf60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70f2e1ae-a69a-4f05-8987-8a66c82acf60" (UID: "70f2e1ae-a69a-4f05-8987-8a66c82acf60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:58.358603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.358570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70f2e1ae-a69a-4f05-8987-8a66c82acf60" (UID: "70f2e1ae-a69a-4f05-8987-8a66c82acf60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:40:58.358705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.358594    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70f2e1ae-a69a-4f05-8987-8a66c82acf60" (UID: "70f2e1ae-a69a-4f05-8987-8a66c82acf60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:40:58.358705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.358622    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f2e1ae-a69a-4f05-8987-8a66c82acf60-kube-api-access-dktcg" (OuterVolumeSpecName: "kube-api-access-dktcg") pod "70f2e1ae-a69a-4f05-8987-8a66c82acf60" (UID: "70f2e1ae-a69a-4f05-8987-8a66c82acf60"). InnerVolumeSpecName "kube-api-access-dktcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:58.458285 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.458243    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.458285 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.458280    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f2e1ae-a69a-4f05-8987-8a66c82acf60-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.458285 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.458290    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktcg\" (UniqueName: \"kubernetes.io/projected/70f2e1ae-a69a-4f05-8987-8a66c82acf60-kube-api-access-dktcg\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.458285 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.458298    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f2e1ae-a69a-4f05-8987-8a66c82acf60-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.623488 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:58.623450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:40:59.017975 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.017880    2561 generic.go:356] "Generic (PLEG): container finished" podID="70f2e1ae-a69a-4f05-8987-8a66c82acf60" containerID="1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea" exitCode=0
Sep 11 12:40:59.017975 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.017924    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"
Sep 11 12:40:59.018463 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.017919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" event={"ID":"70f2e1ae-a69a-4f05-8987-8a66c82acf60","Type":"ContainerDied","Data":"1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea"}
Sep 11 12:40:59.018463 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.018059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m" event={"ID":"70f2e1ae-a69a-4f05-8987-8a66c82acf60","Type":"ContainerDied","Data":"9413779cf4635b78e543cd022d6f4336a4cba4405886a44b58299d6eb03e511b"}
Sep 11 12:40:59.018463 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.018090    2561 scope.go:117] "RemoveContainer" containerID="1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea"
Sep 11 12:40:59.028231 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.028214    2561 scope.go:117] "RemoveContainer" containerID="1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea"
Sep 11 12:40:59.028533 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:40:59.028508    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea\": container with ID starting with 1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea not found: ID does not exist" containerID="1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea"
Sep 11 12:40:59.028633 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.028537    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea"} err="failed to get container status \"1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea\": rpc error: code = NotFound desc = could not find container \"1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea\": container with ID starting with 1c4937e045dd3c900d757b436056b0bbf308c9cf4f9fb59dab191b61e964afea not found: ID does not exist"
Sep 11 12:40:59.039425 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.039384    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"]
Sep 11 12:40:59.043833 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.043812    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cb20c8-6bd5464564-9z26m"]
Sep 11 12:40:59.106652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:40:59.106621    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f2e1ae-a69a-4f05-8987-8a66c82acf60" path="/var/lib/kubelet/pods/70f2e1ae-a69a-4f05-8987-8a66c82acf60/volumes"
Sep 11 12:41:04.243465 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:04.243427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:04.303447 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:04.303409    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:04.494586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:04.490964    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"]
Sep 11 12:41:04.494586 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:04.491208    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy" containerID="cri-o://5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83" gracePeriod=30
Sep 11 12:41:04.496588 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:04.496560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:09.753137 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.753109    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:41:09.871098 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871064    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-data\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871135    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-credential-socket\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-certs\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-podinfo\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvqdn\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-kube-api-access-zvqdn\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-token\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-socket\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-envoy\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871382    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871426    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871465    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istiod-ca-cert\") pod \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\" (UID: \"3f0ede1b-c837-4365-a8ca-b21a5e95bab2\") "
Sep 11 12:41:09.871558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:09.871889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:09.871889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871784    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-data\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.871889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871803    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-credential-socket\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.871889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871820    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.871889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871834    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-workload-socket\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.872159 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.871991    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:41:09.873517 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.873493    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 12:41:09.873640 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.873588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-kube-api-access-zvqdn" (OuterVolumeSpecName: "kube-api-access-zvqdn") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "kube-api-access-zvqdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:09.873891 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.873864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:09.873963 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.873882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f0ede1b-c837-4365-a8ca-b21a5e95bab2" (UID: "3f0ede1b-c837-4365-a8ca-b21a5e95bab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:09.972362 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.972332    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqdn\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-kube-api-access-zvqdn\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.972362 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.972358    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-token\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.972558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.972373    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-envoy\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.972558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.972384    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istiod-ca-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:09.972558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:09.972420    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0ede1b-c837-4365-a8ca-b21a5e95bab2-istio-podinfo\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:41:10.077827 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.077793    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerID="5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83" exitCode=0
Sep 11 12:41:10.078011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.077845    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"
Sep 11 12:41:10.078011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.077846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" event={"ID":"3f0ede1b-c837-4365-a8ca-b21a5e95bab2","Type":"ContainerDied","Data":"5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83"}
Sep 11 12:41:10.078011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.077974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97" event={"ID":"3f0ede1b-c837-4365-a8ca-b21a5e95bab2","Type":"ContainerDied","Data":"ab0d4e52b9b9684f79cd1ae36815c82c9e8247fafc1ca0aca0729b11370d5ff8"}
Sep 11 12:41:10.078011 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.078004    2561 scope.go:117] "RemoveContainer" containerID="5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83"
Sep 11 12:41:10.087453 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.087431    2561 scope.go:117] "RemoveContainer" containerID="9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"
Sep 11 12:41:10.095142 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.095124    2561 scope.go:117] "RemoveContainer" containerID="5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83"
Sep 11 12:41:10.095373 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:41:10.095353    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83\": container with ID starting with 5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83 not found: ID does not exist" containerID="5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83"
Sep 11 12:41:10.095466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.095384    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83"} err="failed to get container status \"5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83\": rpc error: code = NotFound desc = could not find container \"5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83\": container with ID starting with 5e8ea701de05387ce96d5a6f1a5b7ba82a39d5f852f200c7fef018c8bfa1aa83 not found: ID does not exist"
Sep 11 12:41:10.095466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.095422    2561 scope.go:117] "RemoveContainer" containerID="9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"
Sep 11 12:41:10.095706 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:41:10.095689    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7\": container with ID starting with 9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7 not found: ID does not exist" containerID="9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"
Sep 11 12:41:10.095747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.095714    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7"} err="failed to get container status \"9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7\": rpc error: code = NotFound desc = could not find container \"9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7\": container with ID starting with 9eef1953962c201be25aaef5944f8fbee702af43460ac636dc0d39a239ac15c7 not found: ID does not exist"
Sep 11 12:41:10.098780 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.098760    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"]
Sep 11 12:41:10.102329 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.102308    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cb20c8-openshift-default-5fbdfcc5d8-72w97"]
Sep 11 12:41:10.365200 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:10.365153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:11.106112 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:11.106082    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" path="/var/lib/kubelet/pods/3f0ede1b-c837-4365-a8ca-b21a5e95bab2/volumes"
Sep 11 12:41:12.771525 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:12.771490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:14.532166 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:14.532136    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:23.862288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:23.862197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:23.960346 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:23.960314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:24.577083 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:24.577049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:26.885126 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:26.885091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:30.289629 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:30.289588    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:36.312387 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:36.312346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:47.401493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:47.401445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:41:54.200825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:41:54.200787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:42:32.494577 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:42:32.494542    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:42:33.404639 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:42:33.404598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:42:38.457140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:42:38.457106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:42:55.331965 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:42:55.331930    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:16.220783 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:16.220726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:30.116391 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:30.116355    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:31.032072 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:31.032035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:31.500952 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:31.500914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:31.822869 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:31.822774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:31.937197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:31.937164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:34.852612 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:34.852564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:34.952338 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:34.952299    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:36.044409 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.044361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:36.139383 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.139343    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:36.230395 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.230356    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:36.578861 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.578831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:36.679829 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.679793    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:36.707236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707205    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"]
Sep 11 12:43:36.707701 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707682    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e14e285-5c91-43ed-ad46-e83b2e721bb4" containerName="payload-pre-processing"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707703    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e14e285-5c91-43ed-ad46-e83b2e721bb4" containerName="payload-pre-processing"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707727    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707740    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f14a47-9fdc-49c9-b47f-d79105195a32" containerName="maas-api"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707748    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f14a47-9fdc-49c9-b47f-d79105195a32" containerName="maas-api"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707757    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35380b8a-2c03-4cfc-9e92-e5ea9da0f126" containerName="maas-api"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707765    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="35380b8a-2c03-4cfc-9e92-e5ea9da0f126" containerName="maas-api"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707789    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707798    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy"
Sep 11 12:43:36.707811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6a4ec2-a307-48e9-82ce-c294421b1487" containerName="payload-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6a4ec2-a307-48e9-82ce-c294421b1487" containerName="payload-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707830    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707848    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2c5420-3b07-4c24-896f-05094bd13155" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707856    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2c5420-3b07-4c24-896f-05094bd13155" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707871    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b278ca31-662c-43f8-8779-b76ea6ee2f78" containerName="payload-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707879    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b278ca31-662c-43f8-8779-b76ea6ee2f78" containerName="payload-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f2e1ae-a69a-4f05-8987-8a66c82acf60" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707901    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f2e1ae-a69a-4f05-8987-8a66c82acf60" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707911    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707919    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707930    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" containerName="payload-pre-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707939    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" containerName="payload-pre-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3b6a74-ae75-403e-8734-bc7922a704f6" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.707957    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3b6a74-ae75-403e-8734-bc7922a704f6" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708049    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e14e285-5c91-43ed-ad46-e83b2e721bb4" containerName="payload-pre-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708062    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6a4ec2-a307-48e9-82ce-c294421b1487" containerName="payload-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708072    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f2e1ae-a69a-4f05-8987-8a66c82acf60" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708081    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b278ca31-662c-43f8-8779-b76ea6ee2f78" containerName="payload-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708092    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3b6a74-ae75-403e-8734-bc7922a704f6" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="35380b8a-2c03-4cfc-9e92-e5ea9da0f126" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708112    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2c5420-3b07-4c24-896f-05094bd13155" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708120    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f14a47-9fdc-49c9-b47f-d79105195a32" containerName="maas-api"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708129    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708137    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708148    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0ede1b-c837-4365-a8ca-b21a5e95bab2" containerName="istio-proxy"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708156    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4008a2b4-2a13-4f81-a5d2-a3c8d6f4b637" containerName="payload-pre-processing"
Sep 11 12:43:36.708353 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.708165    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3548b9a1-9db3-4b02-a1ce-34064f2407fc" containerName="istio-proxy"
Sep 11 12:43:36.711545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.711525    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:36.714111 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.714089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a1087-dockercfg-5cjtw\""
Sep 11 12:43:36.714217 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.714114    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a1087\""
Sep 11 12:43:36.767544 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.767510    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"]
Sep 11 12:43:36.856189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.856106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qc2\" (UniqueName: \"kubernetes.io/projected/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-kube-api-access-46qc2\") pod \"payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:36.856189 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.856173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:36.957826 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.957778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qc2\" (UniqueName: \"kubernetes.io/projected/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-kube-api-access-46qc2\") pod \"payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:36.958001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.957870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:36.958829 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.958803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:36.965615 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:36.965583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qc2\" (UniqueName: \"kubernetes.io/projected/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-kube-api-access-46qc2\") pod \"payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:37.021598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:37.021563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:37.295972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:37.295941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"]
Sep 11 12:43:37.297559 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:43:37.297532    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1952f4bd_1f50_49f9_ae54_9f2c0243bb13.slice/crio-c807e8f1e31a74adda88f491857a00407b06ae1619656fe5dad90753826ce380 WatchSource:0}: Error finding container c807e8f1e31a74adda88f491857a00407b06ae1619656fe5dad90753826ce380: Status 404 returned error can't find the container with id c807e8f1e31a74adda88f491857a00407b06ae1619656fe5dad90753826ce380
Sep 11 12:43:37.818065 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:37.818027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt" event={"ID":"1952f4bd-1f50-49f9-ae54-9f2c0243bb13","Type":"ContainerStarted","Data":"6e7dadfbd3cad2bab402a0a7752f9ca490b7514d80fbfcafa4388c73c803120e"}
Sep 11 12:43:37.818233 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:37.818085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt" event={"ID":"1952f4bd-1f50-49f9-ae54-9f2c0243bb13","Type":"ContainerStarted","Data":"c807e8f1e31a74adda88f491857a00407b06ae1619656fe5dad90753826ce380"}
Sep 11 12:43:37.818233 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:37.818139    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:37.835442 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:37.835360    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt" podStartSLOduration=1.835346277 podStartE2EDuration="1.835346277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:37.833606597 +0000 UTC m=+1753.224143939" watchObservedRunningTime="2026-09-11 12:43:37.835346277 +0000 UTC m=+1753.225883907"
Sep 11 12:43:44.915356 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:44.915321    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:43:48.825371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:48.825333    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:43:59.881061 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:43:59.881024    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:00.629888 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:00.629822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:00.967698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:00.967612    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:01.447141 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:01.447105    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:03.936017 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:03.935973    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:04.034378 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:04.034346    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:05.239769 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:05.239731    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:05.297674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:05.297632    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:05.796749 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:05.796711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:05.903433 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:05.903382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:05.994775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:05.994739    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:06.041021 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.040987    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"]
Sep 11 12:44:06.044525 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.044507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.046594 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.046571    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d5ae2a\""
Sep 11 12:44:06.046785 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.046645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d5ae2a-dockercfg-4tbqk\""
Sep 11 12:44:06.054141 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.054121    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"]
Sep 11 12:44:06.126282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.126251    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl5h\" (UniqueName: \"kubernetes.io/projected/2155191b-cae0-4fa9-92be-e991e3fff00d-kube-api-access-lcl5h\") pod \"payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.126458 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.126286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2155191b-cae0-4fa9-92be-e991e3fff00d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.227970 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.227924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl5h\" (UniqueName: \"kubernetes.io/projected/2155191b-cae0-4fa9-92be-e991e3fff00d-kube-api-access-lcl5h\") pod \"payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.227970 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.227961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2155191b-cae0-4fa9-92be-e991e3fff00d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.228633 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.228613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2155191b-cae0-4fa9-92be-e991e3fff00d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.235566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.235534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl5h\" (UniqueName: \"kubernetes.io/projected/2155191b-cae0-4fa9-92be-e991e3fff00d-kube-api-access-lcl5h\") pod \"payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.355355 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.355321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.625384 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.625361    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"]
Sep 11 12:44:06.627734 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:44:06.627704    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2155191b_cae0_4fa9_92be_e991e3fff00d.slice/crio-2b1e7f8e30222b4972876359b9bf2a178e686f87c27f504352378a692a8f336c WatchSource:0}: Error finding container 2b1e7f8e30222b4972876359b9bf2a178e686f87c27f504352378a692a8f336c: Status 404 returned error can't find the container with id 2b1e7f8e30222b4972876359b9bf2a178e686f87c27f504352378a692a8f336c
Sep 11 12:44:06.960652 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.960554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d" event={"ID":"2155191b-cae0-4fa9-92be-e991e3fff00d","Type":"ContainerStarted","Data":"9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9"}
Sep 11 12:44:06.960809 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.960720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:06.960809 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.960770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d" event={"ID":"2155191b-cae0-4fa9-92be-e991e3fff00d","Type":"ContainerStarted","Data":"2b1e7f8e30222b4972876359b9bf2a178e686f87c27f504352378a692a8f336c"}
Sep 11 12:44:06.976394 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:06.976344    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d" podStartSLOduration=0.976329822 podStartE2EDuration="976.329822ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:06.974771713 +0000 UTC m=+1782.365309054" watchObservedRunningTime="2026-09-11 12:44:06.976329822 +0000 UTC m=+1782.366867185"
Sep 11 12:44:17.968105 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:17.968024    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:44:29.049076 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:29.049036    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:31.247111 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:31.247075    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:31.304461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:31.304421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:34.521240 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:34.521210    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d_2155191b-cae0-4fa9-92be-e991e3fff00d/payload-processing/0.log"
Sep 11 12:44:38.506969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:38.506932    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:44:44.280579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:44:44.280541    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:00.144582 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:00.144545    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"]
Sep 11 12:45:00.147711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:00.147690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"
Sep 11 12:45:00.151204 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:00.151176    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:45:00.157384 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:00.157249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"]
Sep 11 12:45:00.160070 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:00.160049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"
Sep 11 12:45:00.430989 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:00.430966    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"]
Sep 11 12:45:00.432800 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:45:00.432769    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f3b2dc_c934_428d_81f0_ff084a5a4828.slice/crio-aba510d0931681d66817337e72e0a3c81af67b24ac3c041dc827814acb39f0db WatchSource:0}: Error finding container aba510d0931681d66817337e72e0a3c81af67b24ac3c041dc827814acb39f0db: Status 404 returned error can't find the container with id aba510d0931681d66817337e72e0a3c81af67b24ac3c041dc827814acb39f0db
Sep 11 12:45:01.233617 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:01.233576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9" event={"ID":"a3f3b2dc-c934-428d-81f0-ff084a5a4828","Type":"ContainerStarted","Data":"ae04d5c44f13297688a94dd8dc0d8f6e5470dff27cd373a5a96690a0dddf8a1f"}
Sep 11 12:45:01.234130 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:01.233630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9" event={"ID":"a3f3b2dc-c934-428d-81f0-ff084a5a4828","Type":"ContainerStarted","Data":"aba510d0931681d66817337e72e0a3c81af67b24ac3c041dc827814acb39f0db"}
Sep 11 12:45:01.252499 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:01.252447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9" podStartSLOduration=1.252429258 podStartE2EDuration="1.252429258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:01.250926945 +0000 UTC m=+1836.641464287" watchObservedRunningTime="2026-09-11 12:45:01.252429258 +0000 UTC m=+1836.642966594"
Sep 11 12:45:02.240088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:02.240047    2561 generic.go:356] "Generic (PLEG): container finished" podID="a3f3b2dc-c934-428d-81f0-ff084a5a4828" containerID="ae04d5c44f13297688a94dd8dc0d8f6e5470dff27cd373a5a96690a0dddf8a1f" exitCode=0
Sep 11 12:45:02.240499 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:02.240094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9" event={"ID":"a3f3b2dc-c934-428d-81f0-ff084a5a4828","Type":"ContainerDied","Data":"ae04d5c44f13297688a94dd8dc0d8f6e5470dff27cd373a5a96690a0dddf8a1f"}
Sep 11 12:45:03.378096 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:03.378070    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"
Sep 11 12:45:04.252249 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:04.252220    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9"
Sep 11 12:45:04.252455 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:04.252216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818845-sm2h9" event={"ID":"a3f3b2dc-c934-428d-81f0-ff084a5a4828","Type":"ContainerDied","Data":"aba510d0931681d66817337e72e0a3c81af67b24ac3c041dc827814acb39f0db"}
Sep 11 12:45:04.252455 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:04.252337    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba510d0931681d66817337e72e0a3c81af67b24ac3c041dc827814acb39f0db"
Sep 11 12:45:13.009114 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:13.009073    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:25.323609 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:25.323574    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:25.961309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:25.961273    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d_2155191b-cae0-4fa9-92be-e991e3fff00d/payload-processing/0.log"
Sep 11 12:45:26.225218 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:26.225127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:26.546412 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:26.546308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:26.649023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:26.648971    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:29.526345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:29.526306    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:29.614625 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:29.614594    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:30.785861 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:30.785817    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:30.904926 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:30.904888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:31.297470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.297438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:31.507123 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.507093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:31.603440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.603361    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:45:31.659012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.658979    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"]
Sep 11 12:45:31.659420 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.659394    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f3b2dc-c934-428d-81f0-ff084a5a4828" containerName="cleanup"
Sep 11 12:45:31.659470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.659421    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f3b2dc-c934-428d-81f0-ff084a5a4828" containerName="cleanup"
Sep 11 12:45:31.659510 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.659500    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f3b2dc-c934-428d-81f0-ff084a5a4828" containerName="cleanup"
Sep 11 12:45:31.662564 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.662548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.665230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.665208    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f92540\""
Sep 11 12:45:31.676928 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.676906    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"]
Sep 11 12:45:31.718362 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.718331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f2f5c-60a3-44f9-9030-af0eee04d3b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.718531 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.718383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk9s\" (UniqueName: \"kubernetes.io/projected/229f2f5c-60a3-44f9-9030-af0eee04d3b1-kube-api-access-pfk9s\") pod \"payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.819850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.819807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f2f5c-60a3-44f9-9030-af0eee04d3b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.820236 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.819870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk9s\" (UniqueName: \"kubernetes.io/projected/229f2f5c-60a3-44f9-9030-af0eee04d3b1-kube-api-access-pfk9s\") pod \"payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.820620 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.820599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f2f5c-60a3-44f9-9030-af0eee04d3b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.831331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.831307    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk9s\" (UniqueName: \"kubernetes.io/projected/229f2f5c-60a3-44f9-9030-af0eee04d3b1-kube-api-access-pfk9s\") pod \"payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:31.972951 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:31.972871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:32.252293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:32.252262    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"]
Sep 11 12:45:32.254345 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:45:32.254320    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229f2f5c_60a3_44f9_9030_af0eee04d3b1.slice/crio-ef87762d72d36d54b99fbb174827b7522bfc1ed0219891faba34ec06c1c1fa3d WatchSource:0}: Error finding container ef87762d72d36d54b99fbb174827b7522bfc1ed0219891faba34ec06c1c1fa3d: Status 404 returned error can't find the container with id ef87762d72d36d54b99fbb174827b7522bfc1ed0219891faba34ec06c1c1fa3d
Sep 11 12:45:32.397279 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:32.397236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2" event={"ID":"229f2f5c-60a3-44f9-9030-af0eee04d3b1","Type":"ContainerStarted","Data":"16671b4c15eadbd681dfe388e1fc8b015e8375475bf01463eeee4279f040fb8d"}
Sep 11 12:45:32.397482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:32.397328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2" event={"ID":"229f2f5c-60a3-44f9-9030-af0eee04d3b1","Type":"ContainerStarted","Data":"ef87762d72d36d54b99fbb174827b7522bfc1ed0219891faba34ec06c1c1fa3d"}
Sep 11 12:45:32.397482 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:32.397385    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:32.425371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:32.425318    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2" podStartSLOduration=1.4253025670000001 podStartE2EDuration="1.425302567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:32.425160218 +0000 UTC m=+1867.815697561" watchObservedRunningTime="2026-09-11 12:45:32.425302567 +0000 UTC m=+1867.815839909"
Sep 11 12:45:43.404056 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:43.404021    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:45:54.425131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:45:54.425091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:01.401451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.401410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:01.435494 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.435458    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"]
Sep 11 12:46:01.435999 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.435712    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2" podUID="229f2f5c-60a3-44f9-9030-af0eee04d3b1" containerName="payload-processing" containerID="cri-o://16671b4c15eadbd681dfe388e1fc8b015e8375475bf01463eeee4279f040fb8d" gracePeriod=30
Sep 11 12:46:01.480795 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.480762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:01.534933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.534877    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:01.554769 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.554742    2561 generic.go:356] "Generic (PLEG): container finished" podID="229f2f5c-60a3-44f9-9030-af0eee04d3b1" containerID="16671b4c15eadbd681dfe388e1fc8b015e8375475bf01463eeee4279f040fb8d" exitCode=0
Sep 11 12:46:01.554901 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.554825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2" event={"ID":"229f2f5c-60a3-44f9-9030-af0eee04d3b1","Type":"ContainerDied","Data":"16671b4c15eadbd681dfe388e1fc8b015e8375475bf01463eeee4279f040fb8d"}
Sep 11 12:46:01.624566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.624534    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:01.676794 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.676759    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:01.686823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.686803    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:46:01.699673 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.699645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f2f5c-60a3-44f9-9030-af0eee04d3b1-plugins-config-volume\") pod \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") "
Sep 11 12:46:01.699792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.699777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfk9s\" (UniqueName: \"kubernetes.io/projected/229f2f5c-60a3-44f9-9030-af0eee04d3b1-kube-api-access-pfk9s\") pod \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\" (UID: \"229f2f5c-60a3-44f9-9030-af0eee04d3b1\") "
Sep 11 12:46:01.700001 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.699980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229f2f5c-60a3-44f9-9030-af0eee04d3b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "229f2f5c-60a3-44f9-9030-af0eee04d3b1" (UID: "229f2f5c-60a3-44f9-9030-af0eee04d3b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:46:01.700093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.700061    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229f2f5c-60a3-44f9-9030-af0eee04d3b1-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.701863 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.701843    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229f2f5c-60a3-44f9-9030-af0eee04d3b1-kube-api-access-pfk9s" (OuterVolumeSpecName: "kube-api-access-pfk9s") pod "229f2f5c-60a3-44f9-9030-af0eee04d3b1" (UID: "229f2f5c-60a3-44f9-9030-af0eee04d3b1"). InnerVolumeSpecName "kube-api-access-pfk9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:01.801371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.801334    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfk9s\" (UniqueName: \"kubernetes.io/projected/229f2f5c-60a3-44f9-9030-af0eee04d3b1-kube-api-access-pfk9s\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.986240 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:01.986156    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:02.560740 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:02.560701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2" event={"ID":"229f2f5c-60a3-44f9-9030-af0eee04d3b1","Type":"ContainerDied","Data":"ef87762d72d36d54b99fbb174827b7522bfc1ed0219891faba34ec06c1c1fa3d"}
Sep 11 12:46:02.561152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:02.560760    2561 scope.go:117] "RemoveContainer" containerID="16671b4c15eadbd681dfe388e1fc8b015e8375475bf01463eeee4279f040fb8d"
Sep 11 12:46:02.561152 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:02.560719    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"
Sep 11 12:46:02.592493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:02.592465    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"]
Sep 11 12:46:02.597728 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:02.597707    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f92540-5554644b7-z4cc2"]
Sep 11 12:46:03.107192 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:03.107161    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229f2f5c-60a3-44f9-9030-af0eee04d3b1" path="/var/lib/kubelet/pods/229f2f5c-60a3-44f9-9030-af0eee04d3b1/volumes"
Sep 11 12:46:08.073905 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:08.073868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:08.173879 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:08.173845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:08.346290 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:08.346251    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:12.125875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:12.125833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:27.493763 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.493724    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:27.535846 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.535811    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"]
Sep 11 12:46:27.537128 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.537096    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt" podUID="1952f4bd-1f50-49f9-ae54-9f2c0243bb13" containerName="payload-pre-processing" containerID="cri-o://6e7dadfbd3cad2bab402a0a7752f9ca490b7514d80fbfcafa4388c73c803120e" gracePeriod=30
Sep 11 12:46:27.554272 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.554237    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:27.653789 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.653754    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:27.691225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.691196    2561 generic.go:356] "Generic (PLEG): container finished" podID="1952f4bd-1f50-49f9-ae54-9f2c0243bb13" containerID="6e7dadfbd3cad2bab402a0a7752f9ca490b7514d80fbfcafa4388c73c803120e" exitCode=0
Sep 11 12:46:27.691385 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.691264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt" event={"ID":"1952f4bd-1f50-49f9-ae54-9f2c0243bb13","Type":"ContainerDied","Data":"6e7dadfbd3cad2bab402a0a7752f9ca490b7514d80fbfcafa4388c73c803120e"}
Sep 11 12:46:27.704299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.704262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:27.799616 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.799591    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:46:27.942941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.942907    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46qc2\" (UniqueName: \"kubernetes.io/projected/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-kube-api-access-46qc2\") pod \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") "
Sep 11 12:46:27.943098 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.943028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-plugins-config-volume\") pod \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\" (UID: \"1952f4bd-1f50-49f9-ae54-9f2c0243bb13\") "
Sep 11 12:46:27.943494 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.943469    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1952f4bd-1f50-49f9-ae54-9f2c0243bb13" (UID: "1952f4bd-1f50-49f9-ae54-9f2c0243bb13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:46:27.945038 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:27.945008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-kube-api-access-46qc2" (OuterVolumeSpecName: "kube-api-access-46qc2") pod "1952f4bd-1f50-49f9-ae54-9f2c0243bb13" (UID: "1952f4bd-1f50-49f9-ae54-9f2c0243bb13"). InnerVolumeSpecName "kube-api-access-46qc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:28.044140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.044066    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qc2\" (UniqueName: \"kubernetes.io/projected/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-kube-api-access-46qc2\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:46:28.044140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.044092    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1952f4bd-1f50-49f9-ae54-9f2c0243bb13-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:46:28.105622 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.105577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:28.697277 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.697246    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"
Sep 11 12:46:28.697718 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.697289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt" event={"ID":"1952f4bd-1f50-49f9-ae54-9f2c0243bb13","Type":"ContainerDied","Data":"c807e8f1e31a74adda88f491857a00407b06ae1619656fe5dad90753826ce380"}
Sep 11 12:46:28.697718 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.697345    2561 scope.go:117] "RemoveContainer" containerID="6e7dadfbd3cad2bab402a0a7752f9ca490b7514d80fbfcafa4388c73c803120e"
Sep 11 12:46:28.719228 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.719204    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"]
Sep 11 12:46:28.723997 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:28.723976    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1087-7b4f499d6-7fcxt"]
Sep 11 12:46:29.107012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:29.106983    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1952f4bd-1f50-49f9-ae54-9f2c0243bb13" path="/var/lib/kubelet/pods/1952f4bd-1f50-49f9-ae54-9f2c0243bb13/volumes"
Sep 11 12:46:31.026276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:31.026241    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:31.127355 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:31.127310    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:46:33.191529 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:46:33.191491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:10.336835 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:10.336802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:11.103600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:11.103564    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:17.477714 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:17.477630    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:23.741210 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:23.741168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:23.812276 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:23.812234    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:24.107694 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:24.107656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:24.661903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:24.661868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:35.264386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.264339    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:35.293197 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.293158    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"]
Sep 11 12:47:35.293547 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.293494    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d" podUID="2155191b-cae0-4fa9-92be-e991e3fff00d" containerName="payload-processing" containerID="cri-o://9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9" gracePeriod=30
Sep 11 12:47:35.373860 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.373830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:35.485544 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.485510    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:35.560298 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.560270    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:47:35.642695 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.642651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:35.660380 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.660353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcl5h\" (UniqueName: \"kubernetes.io/projected/2155191b-cae0-4fa9-92be-e991e3fff00d-kube-api-access-lcl5h\") pod \"2155191b-cae0-4fa9-92be-e991e3fff00d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") "
Sep 11 12:47:35.660538 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.660506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2155191b-cae0-4fa9-92be-e991e3fff00d-plugins-config-volume\") pod \"2155191b-cae0-4fa9-92be-e991e3fff00d\" (UID: \"2155191b-cae0-4fa9-92be-e991e3fff00d\") "
Sep 11 12:47:35.660850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.660825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2155191b-cae0-4fa9-92be-e991e3fff00d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2155191b-cae0-4fa9-92be-e991e3fff00d" (UID: "2155191b-cae0-4fa9-92be-e991e3fff00d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:35.662350 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.662328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2155191b-cae0-4fa9-92be-e991e3fff00d-kube-api-access-lcl5h" (OuterVolumeSpecName: "kube-api-access-lcl5h") pod "2155191b-cae0-4fa9-92be-e991e3fff00d" (UID: "2155191b-cae0-4fa9-92be-e991e3fff00d"). InnerVolumeSpecName "kube-api-access-lcl5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:47:35.736086 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.736049    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:35.761351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.761323    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2155191b-cae0-4fa9-92be-e991e3fff00d-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:35.761351 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:35.761346    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcl5h\" (UniqueName: \"kubernetes.io/projected/2155191b-cae0-4fa9-92be-e991e3fff00d-kube-api-access-lcl5h\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:36.030849 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.030811    2561 generic.go:356] "Generic (PLEG): container finished" podID="2155191b-cae0-4fa9-92be-e991e3fff00d" containerID="9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9" exitCode=0
Sep 11 12:47:36.031044 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.030856    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"
Sep 11 12:47:36.031044 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.030883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d" event={"ID":"2155191b-cae0-4fa9-92be-e991e3fff00d","Type":"ContainerDied","Data":"9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9"}
Sep 11 12:47:36.031044 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.030925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d" event={"ID":"2155191b-cae0-4fa9-92be-e991e3fff00d","Type":"ContainerDied","Data":"2b1e7f8e30222b4972876359b9bf2a178e686f87c27f504352378a692a8f336c"}
Sep 11 12:47:36.031044 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.030946    2561 scope.go:117] "RemoveContainer" containerID="9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9"
Sep 11 12:47:36.040336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.040312    2561 scope.go:117] "RemoveContainer" containerID="9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9"
Sep 11 12:47:36.040603 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:47:36.040585    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9\": container with ID starting with 9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9 not found: ID does not exist" containerID="9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9"
Sep 11 12:47:36.040687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.040611    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9"} err="failed to get container status \"9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9\": rpc error: code = NotFound desc = could not find container \"9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9\": container with ID starting with 9cac149271ea8d791f1a70324afd09057afa2b28901e2b24567e7ec433c96bd9 not found: ID does not exist"
Sep 11 12:47:36.054007 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.053984    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"]
Sep 11 12:47:36.057827 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.057806    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5ae2a-7945dcf79f-stt7d"]
Sep 11 12:47:36.149451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:36.149390    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:37.106114 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:37.106082    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2155191b-cae0-4fa9-92be-e991e3fff00d" path="/var/lib/kubelet/pods/2155191b-cae0-4fa9-92be-e991e3fff00d/volumes"
Sep 11 12:47:43.427235 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:43.427195    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:43.504520 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:43.504482    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:43.773473 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:43.773312    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:44.335857 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:44.335821    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:54.612191 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.612150    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"]
Sep 11 12:47:54.612759 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.612440    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" podUID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" containerName="maas-api" containerID="cri-o://0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc" gracePeriod=30
Sep 11 12:47:54.647598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.647565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:54.674209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.674176    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"]
Sep 11 12:47:54.674470 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.674444    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" podUID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" containerName="payload-processing" containerID="cri-o://525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae" gracePeriod=30
Sep 11 12:47:54.745888 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.745853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:54.819880 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.819814    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" podUID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.61:9004: connect: connection refused"
Sep 11 12:47:54.819880 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.819860    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" podUID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": dial tcp 10.133.0.60:8443: connect: connection refused"
Sep 11 12:47:54.843769 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.843650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:54.908015 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.907988    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:54.921472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.921449    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:47:54.961357 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:54.961326    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:47:55.041212 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-openshift-service-ca\") pod \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") "
Sep 11 12:47:55.041212 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt46k\" (UniqueName: \"kubernetes.io/projected/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-kube-api-access-gt46k\") pod \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") "
Sep 11 12:47:55.041491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041273    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-plugins-config-volume\") pod \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\" (UID: \"f9b0f942-5aed-43ad-9eb1-a6a080e43e84\") "
Sep 11 12:47:55.041491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041291    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-metrics-certs\") pod \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") "
Sep 11 12:47:55.041491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-maas-api-tls\") pod \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") "
Sep 11 12:47:55.041491 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84wbz\" (UniqueName: \"kubernetes.io/projected/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-kube-api-access-84wbz\") pod \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\" (UID: \"d89a193a-1f26-48d5-8d03-8c79a60c4d9d\") "
Sep 11 12:47:55.041716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041592    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d89a193a-1f26-48d5-8d03-8c79a60c4d9d" (UID: "d89a193a-1f26-48d5-8d03-8c79a60c4d9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:55.041716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041705    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.041811 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.041699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9b0f942-5aed-43ad-9eb1-a6a080e43e84" (UID: "f9b0f942-5aed-43ad-9eb1-a6a080e43e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:55.043644 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.043603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d89a193a-1f26-48d5-8d03-8c79a60c4d9d" (UID: "d89a193a-1f26-48d5-8d03-8c79a60c4d9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:47:55.043644 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.043621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d89a193a-1f26-48d5-8d03-8c79a60c4d9d" (UID: "d89a193a-1f26-48d5-8d03-8c79a60c4d9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:47:55.043786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.043659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-kube-api-access-gt46k" (OuterVolumeSpecName: "kube-api-access-gt46k") pod "f9b0f942-5aed-43ad-9eb1-a6a080e43e84" (UID: "f9b0f942-5aed-43ad-9eb1-a6a080e43e84"). InnerVolumeSpecName "kube-api-access-gt46k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:47:55.043786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.043680    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-kube-api-access-84wbz" (OuterVolumeSpecName: "kube-api-access-84wbz") pod "d89a193a-1f26-48d5-8d03-8c79a60c4d9d" (UID: "d89a193a-1f26-48d5-8d03-8c79a60c4d9d"). InnerVolumeSpecName "kube-api-access-84wbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:47:55.122328 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.122303    2561 generic.go:356] "Generic (PLEG): container finished" podID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" containerID="525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae" exitCode=0
Sep 11 12:47:55.122461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.122349    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"
Sep 11 12:47:55.122461 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.122387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" event={"ID":"f9b0f942-5aed-43ad-9eb1-a6a080e43e84","Type":"ContainerDied","Data":"525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae"}
Sep 11 12:47:55.122580 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.122460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh" event={"ID":"f9b0f942-5aed-43ad-9eb1-a6a080e43e84","Type":"ContainerDied","Data":"a9d3b93a2ae05200dfc0b81be133d83fe6669e05c2b820ac98c5a84cee929eea"}
Sep 11 12:47:55.122580 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.122490    2561 scope.go:117] "RemoveContainer" containerID="525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae"
Sep 11 12:47:55.123788 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.123766    2561 generic.go:356] "Generic (PLEG): container finished" podID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" containerID="0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc" exitCode=0
Sep 11 12:47:55.123889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.123812    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"
Sep 11 12:47:55.123889 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.123855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" event={"ID":"d89a193a-1f26-48d5-8d03-8c79a60c4d9d","Type":"ContainerDied","Data":"0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc"}
Sep 11 12:47:55.123966 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.123901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f" event={"ID":"d89a193a-1f26-48d5-8d03-8c79a60c4d9d","Type":"ContainerDied","Data":"a0b2de238610498cea435bc6bbed95791b375c7a04c6b0c60f222e01c59f8236"}
Sep 11 12:47:55.131246 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.131231    2561 scope.go:117] "RemoveContainer" containerID="525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae"
Sep 11 12:47:55.131539 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:47:55.131519    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae\": container with ID starting with 525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae not found: ID does not exist" containerID="525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae"
Sep 11 12:47:55.131632 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.131549    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae"} err="failed to get container status \"525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae\": rpc error: code = NotFound desc = could not find container \"525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae\": container with ID starting with 525fc6084e6bfa6251e67186bed5e36b46d2f785a2f25076306b2d799f54d0ae not found: ID does not exist"
Sep 11 12:47:55.131632 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.131572    2561 scope.go:117] "RemoveContainer" containerID="0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc"
Sep 11 12:47:55.140293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.140271    2561 scope.go:117] "RemoveContainer" containerID="0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc"
Sep 11 12:47:55.140624 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:47:55.140599    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc\": container with ID starting with 0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc not found: ID does not exist" containerID="0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc"
Sep 11 12:47:55.140705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.140621    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"]
Sep 11 12:47:55.140705 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.140633    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc"} err="failed to get container status \"0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc\": rpc error: code = NotFound desc = could not find container \"0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc\": container with ID starting with 0f06ace65c0d2dd71f2eb97c8cf98ab2bc1b6f19e6fac8aaeecf6b0cac6b17cc not found: ID does not exist"
Sep 11 12:47:55.142341 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.142314    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.142460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.142342    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.142460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.142358    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.142460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.142370    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wbz\" (UniqueName: \"kubernetes.io/projected/d89a193a-1f26-48d5-8d03-8c79a60c4d9d-kube-api-access-84wbz\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.142460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.142382    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt46k\" (UniqueName: \"kubernetes.io/projected/f9b0f942-5aed-43ad-9eb1-a6a080e43e84-kube-api-access-gt46k\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.144039 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.144018    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aaa824-7c6fb9cc97-px85f"]
Sep 11 12:47:55.155247 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.155223    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"]
Sep 11 12:47:55.157642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.157622    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa824-b5f5869-69frh"]
Sep 11 12:47:55.239093 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:55.239059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:47:57.107066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:57.107033    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" path="/var/lib/kubelet/pods/d89a193a-1f26-48d5-8d03-8c79a60c4d9d/volumes"
Sep 11 12:47:57.107449 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:47:57.107383    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" path="/var/lib/kubelet/pods/f9b0f942-5aed-43ad-9eb1-a6a080e43e84/volumes"
Sep 11 12:48:02.066556 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:02.066521    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:02.172412 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:02.172359    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:02.436269 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:02.436232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:02.442538 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:02.442504    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"]
Sep 11 12:48:02.442734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:02.442708    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy" containerID="cri-o://c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27" gracePeriod=30
Sep 11 12:48:03.048862 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:03.048824    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:07.683102 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.683072    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:48:07.865867 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.865838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-data\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866010 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.865920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4j92\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-kube-api-access-x4j92\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866010 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.865945    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-certs\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866010 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.865963    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-credential-socket\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866010 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.865986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-token\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866010 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-envoy\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866022    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-podinfo\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56bf75-c6aa-4c44-8a45-71c77c116220-istiod-ca-cert\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-socket\") pod \"3c56bf75-c6aa-4c44-8a45-71c77c116220\" (UID: \"3c56bf75-c6aa-4c44-8a45-71c77c116220\") "
Sep 11 12:48:07.866267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866159    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:48:07.866267 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:48:07.866574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:48:07.866574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866438    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.866574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866459    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-data\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.866574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:48:07.866574 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.866516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c56bf75-c6aa-4c44-8a45-71c77c116220-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:48:07.868265 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.868238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:48:07.868265 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.868247    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 12:48:07.868706 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.868687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:48:07.868777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.868729    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-kube-api-access-x4j92" (OuterVolumeSpecName: "kube-api-access-x4j92") pod "3c56bf75-c6aa-4c44-8a45-71c77c116220" (UID: "3c56bf75-c6aa-4c44-8a45-71c77c116220"). InnerVolumeSpecName "kube-api-access-x4j92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:48:07.966866 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966841    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-workload-socket\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.966866 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966866    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4j92\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-kube-api-access-x4j92\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.967030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966876    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-credential-socket\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.967030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966885    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-token\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.967030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966893    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-envoy\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.967030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966901    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c56bf75-c6aa-4c44-8a45-71c77c116220-istio-podinfo\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:07.967030 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:07.966908    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c56bf75-c6aa-4c44-8a45-71c77c116220-istiod-ca-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:08.190422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.190305    2561 generic.go:356] "Generic (PLEG): container finished" podID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerID="c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27" exitCode=0
Sep 11 12:48:08.190422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.190349    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"
Sep 11 12:48:08.190422 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.190361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" event={"ID":"3c56bf75-c6aa-4c44-8a45-71c77c116220","Type":"ContainerDied","Data":"c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27"}
Sep 11 12:48:08.190702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.190429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4" event={"ID":"3c56bf75-c6aa-4c44-8a45-71c77c116220","Type":"ContainerDied","Data":"1228f97f4e5b0e4a5efa8b7834e8d697c5d9c26be25b2c1efdebb10eb3ffd57f"}
Sep 11 12:48:08.190702 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.190460    2561 scope.go:117] "RemoveContainer" containerID="c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27"
Sep 11 12:48:08.199741 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.199719    2561 scope.go:117] "RemoveContainer" containerID="22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"
Sep 11 12:48:08.208257 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.208240    2561 scope.go:117] "RemoveContainer" containerID="c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27"
Sep 11 12:48:08.208598 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:48:08.208581    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27\": container with ID starting with c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27 not found: ID does not exist" containerID="c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27"
Sep 11 12:48:08.208706 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.208605    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27"} err="failed to get container status \"c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27\": rpc error: code = NotFound desc = could not find container \"c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27\": container with ID starting with c2600c393268b0aedc746a2ed0909ba726c8dfdfeaffc269262121694f42bd27 not found: ID does not exist"
Sep 11 12:48:08.208706 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.208620    2561 scope.go:117] "RemoveContainer" containerID="22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"
Sep 11 12:48:08.209022 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:48:08.208993    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874\": container with ID starting with 22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874 not found: ID does not exist" containerID="22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"
Sep 11 12:48:08.209179 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.209051    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874"} err="failed to get container status \"22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874\": rpc error: code = NotFound desc = could not find container \"22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874\": container with ID starting with 22f9a1236ca3653c1f1e07295a3649feb2e43ff50a8932931e7d1350ddc0e874 not found: ID does not exist"
Sep 11 12:48:08.211096 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.211076    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"]
Sep 11 12:48:08.217247 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:08.217223    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa824-openshift-default-76885f59d-9qvx4"]
Sep 11 12:48:09.106601 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:09.106569    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" path="/var/lib/kubelet/pods/3c56bf75-c6aa-4c44-8a45-71c77c116220/volumes"
Sep 11 12:48:13.516578 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.516544    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"]
Sep 11 12:48:13.517015 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.516839    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" podUID="9d3bd457-912c-44a1-83bf-d92a7c84e00a" containerName="maas-api" containerID="cri-o://c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d" gracePeriod=30
Sep 11 12:48:13.557310 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.557276    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:13.622088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.622058    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"]
Sep 11 12:48:13.622295 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.622263    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp" podUID="b5fa7769-cf70-4d8d-b714-44a759196ba2" containerName="payload-processing" containerID="cri-o://37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244" gracePeriod=30
Sep 11 12:48:13.677134 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.677044    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:13.770821 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.770788    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:13.788496 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.788472    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:48:13.906673 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.906646    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:48:13.920203 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3bd457-912c-44a1-83bf-d92a7c84e00a-openshift-service-ca\") pod \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") "
Sep 11 12:48:13.920288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fa7769-cf70-4d8d-b714-44a759196ba2-plugins-config-volume\") pod \"b5fa7769-cf70-4d8d-b714-44a759196ba2\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") "
Sep 11 12:48:13.920288 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2bq5\" (UniqueName: \"kubernetes.io/projected/b5fa7769-cf70-4d8d-b714-44a759196ba2-kube-api-access-t2bq5\") pod \"b5fa7769-cf70-4d8d-b714-44a759196ba2\" (UID: \"b5fa7769-cf70-4d8d-b714-44a759196ba2\") "
Sep 11 12:48:13.920487 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920468    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") pod \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") "
Sep 11 12:48:13.920545 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wxtk\" (UniqueName: \"kubernetes.io/projected/9d3bd457-912c-44a1-83bf-d92a7c84e00a-kube-api-access-5wxtk\") pod \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") "
Sep 11 12:48:13.920603 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-metrics-certs\") pod \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\" (UID: \"9d3bd457-912c-44a1-83bf-d92a7c84e00a\") "
Sep 11 12:48:13.920660 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3bd457-912c-44a1-83bf-d92a7c84e00a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d3bd457-912c-44a1-83bf-d92a7c84e00a" (UID: "9d3bd457-912c-44a1-83bf-d92a7c84e00a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:48:13.920726 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fa7769-cf70-4d8d-b714-44a759196ba2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5fa7769-cf70-4d8d-b714-44a759196ba2" (UID: "b5fa7769-cf70-4d8d-b714-44a759196ba2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:48:13.920875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920855    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3bd457-912c-44a1-83bf-d92a7c84e00a-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:13.920875 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.920874    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5fa7769-cf70-4d8d-b714-44a759196ba2-plugins-config-volume\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:13.922658 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.922631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3bd457-912c-44a1-83bf-d92a7c84e00a-kube-api-access-5wxtk" (OuterVolumeSpecName: "kube-api-access-5wxtk") pod "9d3bd457-912c-44a1-83bf-d92a7c84e00a" (UID: "9d3bd457-912c-44a1-83bf-d92a7c84e00a"). InnerVolumeSpecName "kube-api-access-5wxtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:48:13.922753 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.922734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d3bd457-912c-44a1-83bf-d92a7c84e00a" (UID: "9d3bd457-912c-44a1-83bf-d92a7c84e00a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:48:13.922886 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.922866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d3bd457-912c-44a1-83bf-d92a7c84e00a" (UID: "9d3bd457-912c-44a1-83bf-d92a7c84e00a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:48:13.922941 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.922862    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fa7769-cf70-4d8d-b714-44a759196ba2-kube-api-access-t2bq5" (OuterVolumeSpecName: "kube-api-access-t2bq5") pod "b5fa7769-cf70-4d8d-b714-44a759196ba2" (UID: "b5fa7769-cf70-4d8d-b714-44a759196ba2"). InnerVolumeSpecName "kube-api-access-t2bq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:48:13.944745 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:13.944707    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:14.019183 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.019097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:14.021270 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.021248    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bq5\" (UniqueName: \"kubernetes.io/projected/b5fa7769-cf70-4d8d-b714-44a759196ba2-kube-api-access-t2bq5\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:14.021270 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.021270    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:14.021420 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.021280    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxtk\" (UniqueName: \"kubernetes.io/projected/9d3bd457-912c-44a1-83bf-d92a7c84e00a-kube-api-access-5wxtk\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:14.021420 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.021288    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3bd457-912c-44a1-83bf-d92a7c84e00a-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:14.219050 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.219021    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d3bd457-912c-44a1-83bf-d92a7c84e00a" containerID="c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d" exitCode=0
Sep 11 12:48:14.219222 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.219065    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"
Sep 11 12:48:14.219222 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.219109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" event={"ID":"9d3bd457-912c-44a1-83bf-d92a7c84e00a","Type":"ContainerDied","Data":"c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d"}
Sep 11 12:48:14.219222 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.219158    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh" event={"ID":"9d3bd457-912c-44a1-83bf-d92a7c84e00a","Type":"ContainerDied","Data":"7b9f0bab0c225139cfb0d0a3d375fc5783dc218925da2c71ac477c4b94d727e6"}
Sep 11 12:48:14.219222 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.219189    2561 scope.go:117] "RemoveContainer" containerID="c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d"
Sep 11 12:48:14.220598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.220573    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5fa7769-cf70-4d8d-b714-44a759196ba2" containerID="37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244" exitCode=0
Sep 11 12:48:14.220729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.220618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp" event={"ID":"b5fa7769-cf70-4d8d-b714-44a759196ba2","Type":"ContainerDied","Data":"37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244"}
Sep 11 12:48:14.220729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.220624    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"
Sep 11 12:48:14.220729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.220658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp" event={"ID":"b5fa7769-cf70-4d8d-b714-44a759196ba2","Type":"ContainerDied","Data":"36219aa00af263964d7edb5bb1e65b5a707f5bcbe8ac1c947ce0e1a90940d2ad"}
Sep 11 12:48:14.228600 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.228558    2561 scope.go:117] "RemoveContainer" containerID="c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d"
Sep 11 12:48:14.228909 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:48:14.228884    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d\": container with ID starting with c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d not found: ID does not exist" containerID="c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d"
Sep 11 12:48:14.228972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.228922    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d"} err="failed to get container status \"c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d\": rpc error: code = NotFound desc = could not find container \"c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d\": container with ID starting with c706b1533283bca382380d874b5aa41be03d686e2a0362e5c098782ae641b81d not found: ID does not exist"
Sep 11 12:48:14.228972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.228945    2561 scope.go:117] "RemoveContainer" containerID="37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244"
Sep 11 12:48:14.236826 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.236812    2561 scope.go:117] "RemoveContainer" containerID="37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244"
Sep 11 12:48:14.237095 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:48:14.237079    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244\": container with ID starting with 37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244 not found: ID does not exist" containerID="37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244"
Sep 11 12:48:14.237135 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.237099    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244"} err="failed to get container status \"37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244\": rpc error: code = NotFound desc = could not find container \"37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244\": container with ID starting with 37ed1065b876210cdd7002a0dc4629c570b7cf4151960024504797e82efe8244 not found: ID does not exist"
Sep 11 12:48:14.243186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.243161    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"]
Sep 11 12:48:14.244792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.244758    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f4502d-6f776cc5d6-pjfxh"]
Sep 11 12:48:14.256122 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.256101    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"]
Sep 11 12:48:14.262305 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.262285    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f4502d-7ff65c687f-lfdxp"]
Sep 11 12:48:14.386836 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:14.386802    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:15.108124 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:15.108091    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d3bd457-912c-44a1-83bf-d92a7c84e00a" path="/var/lib/kubelet/pods/9d3bd457-912c-44a1-83bf-d92a7c84e00a/volumes"
Sep 11 12:48:15.108509 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:15.108494    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5fa7769-cf70-4d8d-b714-44a759196ba2" path="/var/lib/kubelet/pods/b5fa7769-cf70-4d8d-b714-44a759196ba2/volumes"
Sep 11 12:48:20.742157 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:20.742115    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:20.845299 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:20.845264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:21.022933 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:21.022848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:23.917798 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:23.917762    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58467f79fd-fp5mk"]
Sep 11 12:48:23.918284 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:23.917982    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58467f79fd-fp5mk" podUID="3c3916d2-4aac-4a68-ab82-f16995fad2f0" containerName="manager" containerID="cri-o://11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec" gracePeriod=30
Sep 11 12:48:24.167932 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.167870    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:48:24.207088 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.207062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-webhook-cert\") pod \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") "
Sep 11 12:48:24.207242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.207126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnkgx\" (UniqueName: \"kubernetes.io/projected/3c3916d2-4aac-4a68-ab82-f16995fad2f0-kube-api-access-xnkgx\") pod \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") "
Sep 11 12:48:24.207242 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.207197    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-metrics-certs\") pod \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\" (UID: \"3c3916d2-4aac-4a68-ab82-f16995fad2f0\") "
Sep 11 12:48:24.209557 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.209528    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3c3916d2-4aac-4a68-ab82-f16995fad2f0" (UID: "3c3916d2-4aac-4a68-ab82-f16995fad2f0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:48:24.209725 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.209694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c3916d2-4aac-4a68-ab82-f16995fad2f0" (UID: "3c3916d2-4aac-4a68-ab82-f16995fad2f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:48:24.209725 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.209706    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3916d2-4aac-4a68-ab82-f16995fad2f0-kube-api-access-xnkgx" (OuterVolumeSpecName: "kube-api-access-xnkgx") pod "3c3916d2-4aac-4a68-ab82-f16995fad2f0" (UID: "3c3916d2-4aac-4a68-ab82-f16995fad2f0"). InnerVolumeSpecName "kube-api-access-xnkgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:48:24.244386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.244353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:48:24.275710 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.275684    2561 generic.go:356] "Generic (PLEG): container finished" podID="3c3916d2-4aac-4a68-ab82-f16995fad2f0" containerID="11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec" exitCode=0
Sep 11 12:48:24.275850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.275722    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-fp5mk"
Sep 11 12:48:24.275850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.275741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-fp5mk" event={"ID":"3c3916d2-4aac-4a68-ab82-f16995fad2f0","Type":"ContainerDied","Data":"11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec"}
Sep 11 12:48:24.275850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.275794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-fp5mk" event={"ID":"3c3916d2-4aac-4a68-ab82-f16995fad2f0","Type":"ContainerDied","Data":"82438ce70670e89de8b985a9951852876e5247eb849220014a81dde1bbbb112b"}
Sep 11 12:48:24.275850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.275824    2561 scope.go:117] "RemoveContainer" containerID="11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec"
Sep 11 12:48:24.284991 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.284974    2561 scope.go:117] "RemoveContainer" containerID="11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec"
Sep 11 12:48:24.285266 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:48:24.285246    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec\": container with ID starting with 11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec not found: ID does not exist" containerID="11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec"
Sep 11 12:48:24.285311 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.285275    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec"} err="failed to get container status \"11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec\": rpc error: code = NotFound desc = could not find container \"11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec\": container with ID starting with 11b04851b5e506d284af87ce1898a1a9a0d44772a970bb855e8595401069d7ec not found: ID does not exist"
Sep 11 12:48:24.296831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.296810    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58467f79fd-fp5mk"]
Sep 11 12:48:24.301185 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.301165    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58467f79fd-fp5mk"]
Sep 11 12:48:24.308729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.308708    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-webhook-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:24.308729 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.308729    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnkgx\" (UniqueName: \"kubernetes.io/projected/3c3916d2-4aac-4a68-ab82-f16995fad2f0-kube-api-access-xnkgx\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:24.308826 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:24.308738    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3916d2-4aac-4a68-ab82-f16995fad2f0-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:48:25.106372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:48:25.106339    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3916d2-4aac-4a68-ab82-f16995fad2f0" path="/var/lib/kubelet/pods/3c3916d2-4aac-4a68-ab82-f16995fad2f0/volumes"
Sep 11 12:49:06.262775 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:06.262719    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:49:06.362947 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:06.362910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:49:25.170563 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170530    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58467f79fd-5twdr"]
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170874    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170886    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170893    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170899    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170909    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" containerName="maas-api"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170915    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" containerName="maas-api"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170921    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2155191b-cae0-4fa9-92be-e991e3fff00d" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170927    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2155191b-cae0-4fa9-92be-e991e3fff00d" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170937    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1952f4bd-1f50-49f9-ae54-9f2c0243bb13" containerName="payload-pre-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170942    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1952f4bd-1f50-49f9-ae54-9f2c0243bb13" containerName="payload-pre-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170947    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fa7769-cf70-4d8d-b714-44a759196ba2" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fa7769-cf70-4d8d-b714-44a759196ba2" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229f2f5c-60a3-44f9-9030-af0eee04d3b1" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170976    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="229f2f5c-60a3-44f9-9030-af0eee04d3b1" containerName="payload-processing"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170986    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3916d2-4aac-4a68-ab82-f16995fad2f0" containerName="manager"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170991    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3916d2-4aac-4a68-ab82-f16995fad2f0" containerName="manager"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.170996    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3bd457-912c-44a1-83bf-d92a7c84e00a" containerName="maas-api"
Sep 11 12:49:25.171012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171001    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3bd457-912c-44a1-83bf-d92a7c84e00a" containerName="maas-api"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171051    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1952f4bd-1f50-49f9-ae54-9f2c0243bb13" containerName="payload-pre-processing"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171060    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fa7769-cf70-4d8d-b714-44a759196ba2" containerName="payload-processing"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171065    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3916d2-4aac-4a68-ab82-f16995fad2f0" containerName="manager"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171071    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171077    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="229f2f5c-60a3-44f9-9030-af0eee04d3b1" containerName="payload-processing"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171084    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2155191b-cae0-4fa9-92be-e991e3fff00d" containerName="payload-processing"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171091    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171097    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3bd457-912c-44a1-83bf-d92a7c84e00a" containerName="maas-api"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171103    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d89a193a-1f26-48d5-8d03-8c79a60c4d9d" containerName="maas-api"
Sep 11 12:49:25.171598 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.171109    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b0f942-5aed-43ad-9eb1-a6a080e43e84" containerName="payload-processing"
Sep 11 12:49:25.174110 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.174092    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.177754 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.177726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-72sgl\""
Sep 11 12:49:25.177893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.177784    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 12:49:25.177893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.177792    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 12:49:25.177893 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.177727    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 12:49:25.185426 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.185390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58467f79fd-5twdr"]
Sep 11 12:49:25.244950 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.244911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-webhook-cert\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.245131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.245062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-metrics-certs\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.245131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.245097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qs6\" (UniqueName: \"kubernetes.io/projected/5e9acacd-1a66-44d6-90b5-6690f55ba82a-kube-api-access-p9qs6\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.346638 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.346602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-metrics-certs\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.346638 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.346638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qs6\" (UniqueName: \"kubernetes.io/projected/5e9acacd-1a66-44d6-90b5-6690f55ba82a-kube-api-access-p9qs6\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.346902 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.346672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-webhook-cert\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.349014 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.348982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-metrics-certs\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.349131 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.349078    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-webhook-cert\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.357310 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.357289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qs6\" (UniqueName: \"kubernetes.io/projected/5e9acacd-1a66-44d6-90b5-6690f55ba82a-kube-api-access-p9qs6\") pod \"maas-controller-58467f79fd-5twdr\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") " pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.484698 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.484610    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:25.781168 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.781141    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58467f79fd-5twdr"]
Sep 11 12:49:25.783742 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:49:25.783703    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9acacd_1a66_44d6_90b5_6690f55ba82a.slice/crio-7ad9c7fc1687efb206a1c945ac8078716314c28ee312f054f3ee4f8b545ac7fa WatchSource:0}: Error finding container 7ad9c7fc1687efb206a1c945ac8078716314c28ee312f054f3ee4f8b545ac7fa: Status 404 returned error can't find the container with id 7ad9c7fc1687efb206a1c945ac8078716314c28ee312f054f3ee4f8b545ac7fa
Sep 11 12:49:25.785031 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:25.785015    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:49:26.572734 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:26.572701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-5twdr" event={"ID":"5e9acacd-1a66-44d6-90b5-6690f55ba82a","Type":"ContainerStarted","Data":"7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b"}
Sep 11 12:49:26.573193 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:26.572752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-5twdr" event={"ID":"5e9acacd-1a66-44d6-90b5-6690f55ba82a","Type":"ContainerStarted","Data":"7ad9c7fc1687efb206a1c945ac8078716314c28ee312f054f3ee4f8b545ac7fa"}
Sep 11 12:49:26.573193 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:26.572842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:37.579067 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:37.579038    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:37.606975 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:37.606930    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58467f79fd-5twdr" podStartSLOduration=11.903739349 podStartE2EDuration="12.606915291s" totalImagesPullingTime="703.175942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:49:26.601506273 +0000 UTC m=+2101.992043614" watchObservedRunningTime="2026-09-11 12:49:37.606915291 +0000 UTC m=+2112.997452633"
Sep 11 12:49:44.533628 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:44.533581    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:49:44.696117 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:44.696083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:49:44.793611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:44.793528    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58467f79fd-5twdr"]
Sep 11 12:49:44.793774 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:44.793748    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58467f79fd-5twdr" podUID="5e9acacd-1a66-44d6-90b5-6690f55ba82a" containerName="manager" containerID="cri-o://7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b" gracePeriod=30
Sep 11 12:49:45.037175 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.037142    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:45.127994 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.127965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qs6\" (UniqueName: \"kubernetes.io/projected/5e9acacd-1a66-44d6-90b5-6690f55ba82a-kube-api-access-p9qs6\") pod \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") "
Sep 11 12:49:45.128193 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.128019    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-webhook-cert\") pod \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") "
Sep 11 12:49:45.128193 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.128072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-metrics-certs\") pod \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\" (UID: \"5e9acacd-1a66-44d6-90b5-6690f55ba82a\") "
Sep 11 12:49:45.130146 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.130115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9acacd-1a66-44d6-90b5-6690f55ba82a-kube-api-access-p9qs6" (OuterVolumeSpecName: "kube-api-access-p9qs6") pod "5e9acacd-1a66-44d6-90b5-6690f55ba82a" (UID: "5e9acacd-1a66-44d6-90b5-6690f55ba82a"). InnerVolumeSpecName "kube-api-access-p9qs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:49:45.130274 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.130152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e9acacd-1a66-44d6-90b5-6690f55ba82a" (UID: "5e9acacd-1a66-44d6-90b5-6690f55ba82a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:49:45.130274 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.130174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5e9acacd-1a66-44d6-90b5-6690f55ba82a" (UID: "5e9acacd-1a66-44d6-90b5-6690f55ba82a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:49:45.228823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.228784    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-webhook-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:49:45.228823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.228816    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9acacd-1a66-44d6-90b5-6690f55ba82a-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:49:45.228823 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.228825    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qs6\" (UniqueName: \"kubernetes.io/projected/5e9acacd-1a66-44d6-90b5-6690f55ba82a-kube-api-access-p9qs6\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:49:45.668677 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.668636    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e9acacd-1a66-44d6-90b5-6690f55ba82a" containerID="7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b" exitCode=0
Sep 11 12:49:45.669155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.668689    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-5twdr"
Sep 11 12:49:45.669155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.668721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-5twdr" event={"ID":"5e9acacd-1a66-44d6-90b5-6690f55ba82a","Type":"ContainerDied","Data":"7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b"}
Sep 11 12:49:45.669155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.668767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-5twdr" event={"ID":"5e9acacd-1a66-44d6-90b5-6690f55ba82a","Type":"ContainerDied","Data":"7ad9c7fc1687efb206a1c945ac8078716314c28ee312f054f3ee4f8b545ac7fa"}
Sep 11 12:49:45.669155 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.668788    2561 scope.go:117] "RemoveContainer" containerID="7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b"
Sep 11 12:49:45.677834 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.677814    2561 scope.go:117] "RemoveContainer" containerID="7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b"
Sep 11 12:49:45.678092 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:49:45.678072    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b\": container with ID starting with 7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b not found: ID does not exist" containerID="7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b"
Sep 11 12:49:45.678133 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.678100    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b"} err="failed to get container status \"7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b\": rpc error: code = NotFound desc = could not find container \"7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b\": container with ID starting with 7bf02918ff34ec02653620d6a480f9e83404b1a84ba83d36b1777098f2cc0e7b not found: ID does not exist"
Sep 11 12:49:45.690834 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.690808    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58467f79fd-5twdr"]
Sep 11 12:49:45.695098 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:45.695078    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58467f79fd-5twdr"]
Sep 11 12:49:46.576818 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.576782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:49:46.687867 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.687837    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58467f79fd-mgksq"]
Sep 11 12:49:46.688229 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.688201    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy"
Sep 11 12:49:46.688229 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.688215    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c56bf75-c6aa-4c44-8a45-71c77c116220" containerName="istio-proxy"
Sep 11 12:49:46.688302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.688243    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9acacd-1a66-44d6-90b5-6690f55ba82a" containerName="manager"
Sep 11 12:49:46.688302 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.688249    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9acacd-1a66-44d6-90b5-6690f55ba82a" containerName="manager"
Sep 11 12:49:46.688371 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.688307    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9acacd-1a66-44d6-90b5-6690f55ba82a" containerName="manager"
Sep 11 12:49:46.692589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.692571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.696163 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.696142    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 12:49:46.696280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.696224    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 12:49:46.696280 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.696260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-72sgl\""
Sep 11 12:49:46.696386 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.696262    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 12:49:46.705261 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.705231    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58467f79fd-mgksq"]
Sep 11 12:49:46.843023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.842940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47ee0d3-8455-48d3-987a-91040db6f3bc-metrics-certs\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.843023 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.842994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrj78\" (UniqueName: \"kubernetes.io/projected/a47ee0d3-8455-48d3-987a-91040db6f3bc-kube-api-access-mrj78\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.843180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.843079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a47ee0d3-8455-48d3-987a-91040db6f3bc-webhook-cert\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.944178 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.944140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj78\" (UniqueName: \"kubernetes.io/projected/a47ee0d3-8455-48d3-987a-91040db6f3bc-kube-api-access-mrj78\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.944345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.944211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a47ee0d3-8455-48d3-987a-91040db6f3bc-webhook-cert\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.944345 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.944267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47ee0d3-8455-48d3-987a-91040db6f3bc-metrics-certs\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.946637 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.946611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a47ee0d3-8455-48d3-987a-91040db6f3bc-webhook-cert\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.946758 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.946713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47ee0d3-8455-48d3-987a-91040db6f3bc-metrics-certs\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:46.952063 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:46.952042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj78\" (UniqueName: \"kubernetes.io/projected/a47ee0d3-8455-48d3-987a-91040db6f3bc-kube-api-access-mrj78\") pod \"maas-controller-58467f79fd-mgksq\" (UID: \"a47ee0d3-8455-48d3-987a-91040db6f3bc\") " pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:47.002481 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.002453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:47.106678 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.106609    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e9acacd-1a66-44d6-90b5-6690f55ba82a" path="/var/lib/kubelet/pods/5e9acacd-1a66-44d6-90b5-6690f55ba82a/volumes"
Sep 11 12:49:47.286610 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.286580    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58467f79fd-mgksq"]
Sep 11 12:49:47.289678 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:49:47.289648    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47ee0d3_8455_48d3_987a_91040db6f3bc.slice/crio-9d272ee5ce292d263ed61a634d713de6db25cb71fa55419654080265434b3d70 WatchSource:0}: Error finding container 9d272ee5ce292d263ed61a634d713de6db25cb71fa55419654080265434b3d70: Status 404 returned error can't find the container with id 9d272ee5ce292d263ed61a634d713de6db25cb71fa55419654080265434b3d70
Sep 11 12:49:47.681014 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.680970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-mgksq" event={"ID":"a47ee0d3-8455-48d3-987a-91040db6f3bc","Type":"ContainerStarted","Data":"67cd8d452656103493be0028b1be2cee39e4c8e6e2ac030c0f4f2c18f224ab5d"}
Sep 11 12:49:47.681171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.681032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58467f79fd-mgksq" event={"ID":"a47ee0d3-8455-48d3-987a-91040db6f3bc","Type":"ContainerStarted","Data":"9d272ee5ce292d263ed61a634d713de6db25cb71fa55419654080265434b3d70"}
Sep 11 12:49:47.681171 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.681085    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:49:47.703747 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:47.703691    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58467f79fd-mgksq" podStartSLOduration=1.382225206 podStartE2EDuration="1.703671248s" totalImagesPullingTime="321.446042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:49:47.701010767 +0000 UTC m=+2123.091548119" watchObservedRunningTime="2026-09-11 12:49:47.703671248 +0000 UTC m=+2123.094208591"
Sep 11 12:49:58.687777 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:49:58.687743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58467f79fd-mgksq"
Sep 11 12:50:22.531725 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:50:22.531643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:51:38.709043 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:51:38.709010    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:52:13.137463 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:13.137425    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:52:19.773942 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:19.773886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:52:35.424602 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:35.424565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:52:52.281827 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.281793    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"]
Sep 11 12:52:52.285870 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.285843    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.288196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.288168    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-614772b9-kserve-self-signed-certs\""
Sep 11 12:52:52.297725 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.297701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"]
Sep 11 12:52:52.310822 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.310787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:52:52.386527 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.386492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-home\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.386687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.386533    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2822ca7-157d-43d6-866e-a601a26c33da-tls-certs\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.386687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.386566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4v6\" (UniqueName: \"kubernetes.io/projected/e2822ca7-157d-43d6-866e-a601a26c33da-kube-api-access-rx4v6\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.386687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.386588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-tmp-dir\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.386687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.386611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-model-cache\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.386687 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.386659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-dshm\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.487813 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.487773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4v6\" (UniqueName: \"kubernetes.io/projected/e2822ca7-157d-43d6-866e-a601a26c33da-kube-api-access-rx4v6\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.487976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.487819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-tmp-dir\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.487976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.487851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-model-cache\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.487976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.487898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-dshm\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.487976 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.487974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-home\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.488167 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.488001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2822ca7-157d-43d6-866e-a601a26c33da-tls-certs\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.488319 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.488296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-tmp-dir\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.488366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.488322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-model-cache\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.488366 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.488352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-home\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.490176 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.490158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-dshm\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.490618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.490596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2822ca7-157d-43d6-866e-a601a26c33da-tls-certs\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.496985 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.496960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4v6\" (UniqueName: \"kubernetes.io/projected/e2822ca7-157d-43d6-866e-a601a26c33da-kube-api-access-rx4v6\") pod \"e2e-same-id-b-614772b9-kserve-6998564899-66gdt\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") " pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.596909 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.596875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:52:52.866611 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:52.866591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"]
Sep 11 12:52:52.868810 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:52:52.868778    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2822ca7_157d_43d6_866e_a601a26c33da.slice/crio-743bedd3703c1cc34fe28374d6f26751648f58668ce2afc51cf41c97e249deba WatchSource:0}: Error finding container 743bedd3703c1cc34fe28374d6f26751648f58668ce2afc51cf41c97e249deba: Status 404 returned error can't find the container with id 743bedd3703c1cc34fe28374d6f26751648f58668ce2afc51cf41c97e249deba
Sep 11 12:52:53.535209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:53.535172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" event={"ID":"e2822ca7-157d-43d6-866e-a601a26c33da","Type":"ContainerStarted","Data":"0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034"}
Sep 11 12:52:53.535619 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:53.535217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" event={"ID":"e2822ca7-157d-43d6-866e-a601a26c33da","Type":"ContainerStarted","Data":"743bedd3703c1cc34fe28374d6f26751648f58668ce2afc51cf41c97e249deba"}
Sep 11 12:52:53.553580 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:52:53.553525    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" podStartSLOduration=1.553507487 podStartE2EDuration="1.553507487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:52:53.552453802 +0000 UTC m=+2308.942991145" watchObservedRunningTime="2026-09-11 12:52:53.553507487 +0000 UTC m=+2308.944044829"
Sep 11 12:53:02.597983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:02.597946    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:53:02.598395 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:02.597996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:53:02.610523 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:02.610497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:53:03.590930 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:03.590897    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:53:16.726608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:16.726565    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:53:18.443642 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:18.443546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"]
Sep 11 12:53:18.457915 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:18.457885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:53:33.463307 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.463268    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" podUID="e2822ca7-157d-43d6-866e-a601a26c33da" containerName="main" containerID="cri-o://0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034" gracePeriod=45
Sep 11 12:53:33.580255 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.580212    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" podUID="e2822ca7-157d-43d6-866e-a601a26c33da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 11 12:53:33.722627 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.722568    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:53:33.723066 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.723046    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2822ca7-157d-43d6-866e-a601a26c33da" containerID="0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034" exitCode=0
Sep 11 12:53:33.723140 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.723117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" event={"ID":"e2822ca7-157d-43d6-866e-a601a26c33da","Type":"ContainerDied","Data":"0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034"}
Sep 11 12:53:33.723186 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.723166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt" event={"ID":"e2822ca7-157d-43d6-866e-a601a26c33da","Type":"ContainerDied","Data":"743bedd3703c1cc34fe28374d6f26751648f58668ce2afc51cf41c97e249deba"}
Sep 11 12:53:33.723222 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.723186    2561 scope.go:117] "RemoveContainer" containerID="0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034"
Sep 11 12:53:33.731209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.731194    2561 scope.go:117] "RemoveContainer" containerID="0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034"
Sep 11 12:53:33.731502 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:53:33.731483    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034\": container with ID starting with 0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034 not found: ID does not exist" containerID="0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034"
Sep 11 12:53:33.731579 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.731511    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034"} err="failed to get container status \"0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034\": rpc error: code = NotFound desc = could not find container \"0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034\": container with ID starting with 0ea1609615bf97269ca5de75d750e7044e1d594c518672a316b338f88098a034 not found: ID does not exist"
Sep 11 12:53:33.862983 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.862953    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-home\") pod \"e2822ca7-157d-43d6-866e-a601a26c33da\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") "
Sep 11 12:53:33.863150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx4v6\" (UniqueName: \"kubernetes.io/projected/e2822ca7-157d-43d6-866e-a601a26c33da-kube-api-access-rx4v6\") pod \"e2822ca7-157d-43d6-866e-a601a26c33da\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") "
Sep 11 12:53:33.863150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-dshm\") pod \"e2822ca7-157d-43d6-866e-a601a26c33da\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") "
Sep 11 12:53:33.863150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-tmp-dir\") pod \"e2822ca7-157d-43d6-866e-a601a26c33da\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") "
Sep 11 12:53:33.863150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-model-cache\") pod \"e2822ca7-157d-43d6-866e-a601a26c33da\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") "
Sep 11 12:53:33.863150 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863146    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2822ca7-157d-43d6-866e-a601a26c33da-tls-certs\") pod \"e2822ca7-157d-43d6-866e-a601a26c33da\" (UID: \"e2822ca7-157d-43d6-866e-a601a26c33da\") "
Sep 11 12:53:33.863440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863247    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-home" (OuterVolumeSpecName: "home") pod "e2822ca7-157d-43d6-866e-a601a26c33da" (UID: "e2822ca7-157d-43d6-866e-a601a26c33da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:33.863440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863327    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2822ca7-157d-43d6-866e-a601a26c33da" (UID: "e2822ca7-157d-43d6-866e-a601a26c33da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:33.863440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863423    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-home\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:33.863558 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.863437    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2822ca7-157d-43d6-866e-a601a26c33da" (UID: "e2822ca7-157d-43d6-866e-a601a26c33da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:33.865378 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.865347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2822ca7-157d-43d6-866e-a601a26c33da-kube-api-access-rx4v6" (OuterVolumeSpecName: "kube-api-access-rx4v6") pod "e2822ca7-157d-43d6-866e-a601a26c33da" (UID: "e2822ca7-157d-43d6-866e-a601a26c33da"). InnerVolumeSpecName "kube-api-access-rx4v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:53:33.865529 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.865370    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2822ca7-157d-43d6-866e-a601a26c33da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2822ca7-157d-43d6-866e-a601a26c33da" (UID: "e2822ca7-157d-43d6-866e-a601a26c33da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:33.865529 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.865421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-dshm" (OuterVolumeSpecName: "dshm") pod "e2822ca7-157d-43d6-866e-a601a26c33da" (UID: "e2822ca7-157d-43d6-866e-a601a26c33da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:33.964711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.964678    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2822ca7-157d-43d6-866e-a601a26c33da-tls-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:33.964711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.964705    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4v6\" (UniqueName: \"kubernetes.io/projected/e2822ca7-157d-43d6-866e-a601a26c33da-kube-api-access-rx4v6\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:33.964711 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.964716    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-dshm\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:33.964931 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.964726    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-tmp-dir\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:33.964931 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:33.964735    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2822ca7-157d-43d6-866e-a601a26c33da-model-cache\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:34.728910 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:34.728878    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"
Sep 11 12:53:34.748486 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:34.748460    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"]
Sep 11 12:53:34.754080 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:34.754051    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-614772b9-kserve-6998564899-66gdt"]
Sep 11 12:53:35.106523 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.106493    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2822ca7-157d-43d6-866e-a601a26c33da" path="/var/lib/kubelet/pods/e2822ca7-157d-43d6-866e-a601a26c33da/volumes"
Sep 11 12:53:35.434848 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.434773    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"]
Sep 11 12:53:35.435309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.435282    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2822ca7-157d-43d6-866e-a601a26c33da" containerName="main"
Sep 11 12:53:35.435309 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.435310    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2822ca7-157d-43d6-866e-a601a26c33da" containerName="main"
Sep 11 12:53:35.435503 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.435434    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2822ca7-157d-43d6-866e-a601a26c33da" containerName="main"
Sep 11 12:53:35.439816 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.439799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.447608 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.447538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"]
Sep 11 12:53:35.578786 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.578751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-maas-api-tls\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.578957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.578794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8hc2\" (UniqueName: \"kubernetes.io/projected/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-kube-api-access-c8hc2\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.578957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.578831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-metrics-certs\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.578957 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.578856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-openshift-service-ca\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.679792 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.679756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-maas-api-tls\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.679953 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.679800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8hc2\" (UniqueName: \"kubernetes.io/projected/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-kube-api-access-c8hc2\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.679953 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.679927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-metrics-certs\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.680043 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.679971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-openshift-service-ca\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.680596 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.680569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-openshift-service-ca\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.682282 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.682259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-maas-api-tls\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.682439 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.682389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-metrics-certs\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.690183 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.690108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8hc2\" (UniqueName: \"kubernetes.io/projected/6510af2d-4487-4c32-b8c7-c6277dbcbc3e-kube-api-access-c8hc2\") pod \"maas-api-57b85b5ff7-4vggw\" (UID: \"6510af2d-4487-4c32-b8c7-c6277dbcbc3e\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:35.752196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:35.752170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:36.027100 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:36.027076    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"]
Sep 11 12:53:36.029678 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:53:36.029650    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6510af2d_4487_4c32_b8c7_c6277dbcbc3e.slice/crio-64d208b575761e6d1e2eb737313f7804ec9a212037ef0ff70bae9c9498565f8e WatchSource:0}: Error finding container 64d208b575761e6d1e2eb737313f7804ec9a212037ef0ff70bae9c9498565f8e: Status 404 returned error can't find the container with id 64d208b575761e6d1e2eb737313f7804ec9a212037ef0ff70bae9c9498565f8e
Sep 11 12:53:36.740903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:36.740804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw" event={"ID":"6510af2d-4487-4c32-b8c7-c6277dbcbc3e","Type":"ContainerStarted","Data":"32aeb84a1daf6dcb79c4ab92010e58669245d02f0a594412309e976588ef5172"}
Sep 11 12:53:36.740903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:36.740870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw" event={"ID":"6510af2d-4487-4c32-b8c7-c6277dbcbc3e","Type":"ContainerStarted","Data":"64d208b575761e6d1e2eb737313f7804ec9a212037ef0ff70bae9c9498565f8e"}
Sep 11 12:53:36.740903 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:36.740898    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:36.759859 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:36.759810    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw" podStartSLOduration=1.43528118 podStartE2EDuration="1.759795019s" totalImagesPullingTime="324.513839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:53:36.757880301 +0000 UTC m=+2352.148417643" watchObservedRunningTime="2026-09-11 12:53:36.759795019 +0000 UTC m=+2352.150332401"
Sep 11 12:53:42.751636 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:42.751605    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-4vggw"
Sep 11 12:53:50.098798 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:50.098758    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"]
Sep 11 12:53:50.099245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:50.099000    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" podUID="8981219b-7b2e-4505-a669-dbbe1473c69f" containerName="maas-api" containerID="cri-o://fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92" gracePeriod=30
Sep 11 12:53:50.801372 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:50.801326    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" podUID="8981219b-7b2e-4505-a669-dbbe1473c69f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.32:8443/health\": dial tcp 10.133.0.32:8443: connect: connection refused"
Sep 11 12:53:51.344187 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.344160    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:53:51.409138 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.409112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs\") pod \"8981219b-7b2e-4505-a669-dbbe1473c69f\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") "
Sep 11 12:53:51.409298 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.409159    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-maas-api-tls\") pod \"8981219b-7b2e-4505-a669-dbbe1473c69f\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") "
Sep 11 12:53:51.409298 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.409192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcsv4\" (UniqueName: \"kubernetes.io/projected/8981219b-7b2e-4505-a669-dbbe1473c69f-kube-api-access-lcsv4\") pod \"8981219b-7b2e-4505-a669-dbbe1473c69f\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") "
Sep 11 12:53:51.409298 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.409249    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8981219b-7b2e-4505-a669-dbbe1473c69f-openshift-service-ca\") pod \"8981219b-7b2e-4505-a669-dbbe1473c69f\" (UID: \"8981219b-7b2e-4505-a669-dbbe1473c69f\") "
Sep 11 12:53:51.409674 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.409646    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8981219b-7b2e-4505-a669-dbbe1473c69f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8981219b-7b2e-4505-a669-dbbe1473c69f" (UID: "8981219b-7b2e-4505-a669-dbbe1473c69f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:53:51.411178 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.411152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8981219b-7b2e-4505-a669-dbbe1473c69f" (UID: "8981219b-7b2e-4505-a669-dbbe1473c69f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:51.411271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.411175    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8981219b-7b2e-4505-a669-dbbe1473c69f" (UID: "8981219b-7b2e-4505-a669-dbbe1473c69f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:51.411271 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.411256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8981219b-7b2e-4505-a669-dbbe1473c69f-kube-api-access-lcsv4" (OuterVolumeSpecName: "kube-api-access-lcsv4") pod "8981219b-7b2e-4505-a669-dbbe1473c69f" (UID: "8981219b-7b2e-4505-a669-dbbe1473c69f"). InnerVolumeSpecName "kube-api-access-lcsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:53:51.510364 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.510291    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-metrics-certs\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:51.510364 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.510317    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8981219b-7b2e-4505-a669-dbbe1473c69f-maas-api-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:51.510364 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.510332    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcsv4\" (UniqueName: \"kubernetes.io/projected/8981219b-7b2e-4505-a669-dbbe1473c69f-kube-api-access-lcsv4\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:51.510364 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.510346    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8981219b-7b2e-4505-a669-dbbe1473c69f-openshift-service-ca\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 12:53:51.816905 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.816820    2561 generic.go:356] "Generic (PLEG): container finished" podID="8981219b-7b2e-4505-a669-dbbe1473c69f" containerID="fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92" exitCode=0
Sep 11 12:53:51.816905 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.816865    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"
Sep 11 12:53:51.816905 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.816877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" event={"ID":"8981219b-7b2e-4505-a669-dbbe1473c69f","Type":"ContainerDied","Data":"fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92"}
Sep 11 12:53:51.817119 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.816930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-cx8jp" event={"ID":"8981219b-7b2e-4505-a669-dbbe1473c69f","Type":"ContainerDied","Data":"33a751440811160bfeaffb6b2a8710dbae0bcdbfe85750a934374d56025803df"}
Sep 11 12:53:51.817119 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.816960    2561 scope.go:117] "RemoveContainer" containerID="fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92"
Sep 11 12:53:51.826209 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.826190    2561 scope.go:117] "RemoveContainer" containerID="fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92"
Sep 11 12:53:51.826486 ip-10-0-135-5 kubenswrapper[2561]: E0911 12:53:51.826463    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92\": container with ID starting with fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92 not found: ID does not exist" containerID="fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92"
Sep 11 12:53:51.826589 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.826492    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92"} err="failed to get container status \"fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92\": rpc error: code = NotFound desc = could not find container \"fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92\": container with ID starting with fe614c9bbae0fd1c2a35ec0b77c6845fb04a403e92cbfe27980778eceb2d5d92 not found: ID does not exist"
Sep 11 12:53:51.837831 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.837802    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"]
Sep 11 12:53:51.839663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:51.839645    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-cx8jp"]
Sep 11 12:53:53.106270 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:53:53.106237    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8981219b-7b2e-4505-a669-dbbe1473c69f" path="/var/lib/kubelet/pods/8981219b-7b2e-4505-a669-dbbe1473c69f/volumes"
Sep 11 12:54:08.551913 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:08.551878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jhw8r_a7952582-3f47-43c5-99c9-c9f711169df2/manager/0.log"
Sep 11 12:54:08.677599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:08.677566    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nwltg_7eba0dae-21aa-4501-9a7c-1ff4542a2609/manager/0.log"
Sep 11 12:54:08.812885 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:08.812809    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58467f79fd-mgksq_a47ee0d3-8455-48d3-987a-91040db6f3bc/manager/0.log"
Sep 11 12:54:09.178293 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:09.178267    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-k56gw_e3686c6d-9ad6-4114-ac21-c629251fd600/copy-manifests/0.log"
Sep 11 12:54:09.185548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:09.185524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-k56gw_e3686c6d-9ad6-4114-ac21-c629251fd600/manager/0.log"
Sep 11 12:54:09.757019 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:09.756980    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57b85b5ff7-4vggw_6510af2d-4487-4c32-b8c7-c6277dbcbc3e/maas-api/0.log"
Sep 11 12:54:10.072004 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:10.071920    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818845-sm2h9_a3f3b2dc-c934-428d-81f0-ff084a5a4828/cleanup/0.log"
Sep 11 12:54:10.180850 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:10.180821    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zz4sn_4df10631-9cd8-4c01-9e26-0b5dadb94ac7/postgres/0.log"
Sep 11 12:54:11.151196 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:11.151161    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jxx8m_6a4583d8-7f62-401f-aba6-f36cf834061d/manager/0.log"
Sep 11 12:54:11.638329 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:11.638295    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vq5jn_a632f042-67c1-48ee-9e1c-06bb2a196cd7/manager/0.log"
Sep 11 12:54:12.581076 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:12.581045    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lb4vt_1446188b-058f-4d03-8064-059b91eedfe7/payload-pre-processing/0.log"
Sep 11 12:54:13.150118 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:13.150087    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-brk64_1c425ab8-bbac-4bc9-ac77-419f4e5c10f3/main/0.log"
Sep 11 12:54:13.262681 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:13.262654    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fd7mf_92577119-65c3-41bf-b42e-91a5c174ef1a/main/0.log"
Sep 11 12:54:19.161602 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.161566    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkgjl/must-gather-mpfqv"]
Sep 11 12:54:19.162070 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.161939    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8981219b-7b2e-4505-a669-dbbe1473c69f" containerName="maas-api"
Sep 11 12:54:19.162070 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.161950    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8981219b-7b2e-4505-a669-dbbe1473c69f" containerName="maas-api"
Sep 11 12:54:19.162070 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.162008    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8981219b-7b2e-4505-a669-dbbe1473c69f" containerName="maas-api"
Sep 11 12:54:19.165225 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.165201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.168012 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.167990    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkgjl\"/\"openshift-service-ca.crt\""
Sep 11 12:54:19.168230 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.168214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkgjl\"/\"kube-root-ca.crt\""
Sep 11 12:54:19.175161 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.175139    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/must-gather-mpfqv"]
Sep 11 12:54:19.177460 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.177435    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:54:19.234944 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.234898    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:54:19.240953 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.240930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b349ad4-8577-449c-bd6f-ac526bf4bccb-must-gather-output\") pod \"must-gather-mpfqv\" (UID: \"8b349ad4-8577-449c-bd6f-ac526bf4bccb\") " pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.241064 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.241030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25n4b\" (UniqueName: \"kubernetes.io/projected/8b349ad4-8577-449c-bd6f-ac526bf4bccb-kube-api-access-25n4b\") pod \"must-gather-mpfqv\" (UID: \"8b349ad4-8577-449c-bd6f-ac526bf4bccb\") " pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.342121 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.342090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n4b\" (UniqueName: \"kubernetes.io/projected/8b349ad4-8577-449c-bd6f-ac526bf4bccb-kube-api-access-25n4b\") pod \"must-gather-mpfqv\" (UID: \"8b349ad4-8577-449c-bd6f-ac526bf4bccb\") " pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.342245 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.342135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b349ad4-8577-449c-bd6f-ac526bf4bccb-must-gather-output\") pod \"must-gather-mpfqv\" (UID: \"8b349ad4-8577-449c-bd6f-ac526bf4bccb\") " pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.342466 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.342450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b349ad4-8577-449c-bd6f-ac526bf4bccb-must-gather-output\") pod \"must-gather-mpfqv\" (UID: \"8b349ad4-8577-449c-bd6f-ac526bf4bccb\") " pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.349808 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.349782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n4b\" (UniqueName: \"kubernetes.io/projected/8b349ad4-8577-449c-bd6f-ac526bf4bccb-kube-api-access-25n4b\") pod \"must-gather-mpfqv\" (UID: \"8b349ad4-8577-449c-bd6f-ac526bf4bccb\") " pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.474630 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.474557    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/must-gather-mpfqv"
Sep 11 12:54:19.746980 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.746877    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/must-gather-mpfqv"]
Sep 11 12:54:19.756147 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:54:19.756105    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b349ad4_8577_449c_bd6f_ac526bf4bccb.slice/crio-704b9c7b30987cc4a038d6c4e65f8ada685721e7bdbf24c0db08305cb8af8dc2 WatchSource:0}: Error finding container 704b9c7b30987cc4a038d6c4e65f8ada685721e7bdbf24c0db08305cb8af8dc2: Status 404 returned error can't find the container with id 704b9c7b30987cc4a038d6c4e65f8ada685721e7bdbf24c0db08305cb8af8dc2
Sep 11 12:54:19.949440 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:19.949386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/must-gather-mpfqv" event={"ID":"8b349ad4-8577-449c-bd6f-ac526bf4bccb","Type":"ContainerStarted","Data":"704b9c7b30987cc4a038d6c4e65f8ada685721e7bdbf24c0db08305cb8af8dc2"}
Sep 11 12:54:20.958115 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:20.958075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/must-gather-mpfqv" event={"ID":"8b349ad4-8577-449c-bd6f-ac526bf4bccb","Type":"ContainerStarted","Data":"f3b829cff0a5eba16b6de41be2d92f326b9b4f8d36a007b9f89ef38325bb3d08"}
Sep 11 12:54:20.958599 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:20.958131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/must-gather-mpfqv" event={"ID":"8b349ad4-8577-449c-bd6f-ac526bf4bccb","Type":"ContainerStarted","Data":"303821c942797dea39188a410f4c25b54e02e1406461ad9b5d10f04a216574d1"}
Sep 11 12:54:20.976127 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:20.976076    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkgjl/must-gather-mpfqv" podStartSLOduration=1.347051077 podStartE2EDuration="1.976059702s" totalImagesPullingTime="629.008625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:54:20.974068596 +0000 UTC m=+2396.364605939" watchObservedRunningTime="2026-09-11 12:54:20.976059702 +0000 UTC m=+2396.366597045"
Sep 11 12:54:21.666452 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:21.666410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:54:21.923051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:21.922971    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2cbzp_87a282b2-6cba-47e6-8831-de744de9a5af/global-pull-secret-syncer/0.log"
Sep 11 12:54:22.088051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:22.088013    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log"
Sep 11 12:54:22.214778 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:22.214701    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-5.ec2.internal_243f33b0da41e521566de787222b729f/haproxy/0.log"
Sep 11 12:54:27.076534 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:27.076500    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jxx8m_6a4583d8-7f62-401f-aba6-f36cf834061d/manager/0.log"
Sep 11 12:54:27.266571 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:27.266525    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vq5jn_a632f042-67c1-48ee-9e1c-06bb2a196cd7/manager/0.log"
Sep 11 12:54:28.703845 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.703809    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/alertmanager/0.log"
Sep 11 12:54:28.740331 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.740301    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/config-reloader/0.log"
Sep 11 12:54:28.773651 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.773611    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/kube-rbac-proxy-web/0.log"
Sep 11 12:54:28.803631 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.803599    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/kube-rbac-proxy/0.log"
Sep 11 12:54:28.828548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.828515    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/kube-rbac-proxy-metric/0.log"
Sep 11 12:54:28.853979 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.853947    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/prom-label-proxy/0.log"
Sep 11 12:54:28.890826 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:28.890799    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ba64768a-0450-4576-ac47-21e2061ac093/init-config-reloader/0.log"
Sep 11 12:54:29.092956 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.092922    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4mqfw_9115c913-bc58-4ded-a06f-53c8c67feb43/node-exporter/0.log"
Sep 11 12:54:29.138317 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.138284    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4mqfw_9115c913-bc58-4ded-a06f-53c8c67feb43/kube-rbac-proxy/0.log"
Sep 11 12:54:29.173055 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.171764    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4mqfw_9115c913-bc58-4ded-a06f-53c8c67feb43/init-textfile/0.log"
Sep 11 12:54:29.396151 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.396055    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mqm5b_4a0ba35d-63b1-46b1-adae-b265d3534b9c/kube-rbac-proxy-main/0.log"
Sep 11 12:54:29.420836 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.420811    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mqm5b_4a0ba35d-63b1-46b1-adae-b265d3534b9c/kube-rbac-proxy-self/0.log"
Sep 11 12:54:29.452388 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.452357    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mqm5b_4a0ba35d-63b1-46b1-adae-b265d3534b9c/openshift-state-metrics/0.log"
Sep 11 12:54:29.494916 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.494857    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/prometheus/0.log"
Sep 11 12:54:29.550180 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.550146    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/config-reloader/0.log"
Sep 11 12:54:29.580451 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.580420    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/thanos-sidecar/0.log"
Sep 11 12:54:29.607342 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.607311    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/kube-rbac-proxy-web/0.log"
Sep 11 12:54:29.631830 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.631792    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/kube-rbac-proxy/0.log"
Sep 11 12:54:29.654633 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.654559    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/kube-rbac-proxy-thanos/0.log"
Sep 11 12:54:29.683837 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.683805    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36f90cee-68d3-48ff-b50f-6a8565838916/init-config-reloader/0.log"
Sep 11 12:54:29.769736 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:29.769704    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-9htxm_811275af-c417-4c05-b143-ce812ea8d206/prometheus-operator-admission-webhook/0.log"
Sep 11 12:54:32.319726 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.319685    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"]
Sep 11 12:54:32.327292 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.327258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.329972 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.329946    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zkgjl\"/\"default-dockercfg-dl5jf\""
Sep 11 12:54:32.344730 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.344699    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"]
Sep 11 12:54:32.494548 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.494509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-proc\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.494716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.494561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-lib-modules\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.494716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.494646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-sys\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.494716 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.494685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv95r\" (UniqueName: \"kubernetes.io/projected/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-kube-api-access-mv95r\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.494839 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.494720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-podres\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596040 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.595998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-podres\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596326 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-podres\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-proc\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596492 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-lib-modules\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-sys\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596618 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv95r\" (UniqueName: \"kubernetes.io/projected/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-kube-api-access-mv95r\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596817 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-proc\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.596932 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-lib-modules\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.597005 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.596911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-sys\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.607156 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.606981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv95r\" (UniqueName: \"kubernetes.io/projected/1442abc9-98dd-47e0-ab88-b97ec5db3bb1-kube-api-access-mv95r\") pod \"perf-node-gather-daemonset-r54n6\" (UID: \"1442abc9-98dd-47e0-ab88-b97ec5db3bb1\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.639664 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.639630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:32.918117 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.918090    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"]
Sep 11 12:54:32.920474 ip-10-0-135-5 kubenswrapper[2561]: W0911 12:54:32.920443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1442abc9_98dd_47e0_ab88_b97ec5db3bb1.slice/crio-39c12591841ae3caa6bc3a2efa9d1e48bd5970d958daf20ca5c713fb64f2605a WatchSource:0}: Error finding container 39c12591841ae3caa6bc3a2efa9d1e48bd5970d958daf20ca5c713fb64f2605a: Status 404 returned error can't find the container with id 39c12591841ae3caa6bc3a2efa9d1e48bd5970d958daf20ca5c713fb64f2605a
Sep 11 12:54:32.996110 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:32.996060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c6rx_23350a94-a70f-4322-8fad-a5ddf5152cf4/konnectivity-agent/0.log" limit=1024
Sep 11 12:54:33.050743 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:33.050705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6" event={"ID":"1442abc9-98dd-47e0-ab88-b97ec5db3bb1","Type":"ContainerStarted","Data":"39c12591841ae3caa6bc3a2efa9d1e48bd5970d958daf20ca5c713fb64f2605a"}
Sep 11 12:54:34.057568 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:34.057517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6" event={"ID":"1442abc9-98dd-47e0-ab88-b97ec5db3bb1","Type":"ContainerStarted","Data":"79612d0f81f50a91d7489fbb5a633476545733b8b0f71f7003583c411c3fdd9b"}
Sep 11 12:54:34.057999 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:34.057704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:34.080062 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:34.080011    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6" podStartSLOduration=2.079995531 podStartE2EDuration="2.079995531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:54:34.077159642 +0000 UTC m=+2409.467696984" watchObservedRunningTime="2026-09-11 12:54:34.079995531 +0000 UTC m=+2409.470532872"
Sep 11 12:54:34.453336 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:34.453310    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2rq7_8f3f287d-baa3-45f9-9108-429e7bf73b9a/dns/0.log"
Sep 11 12:54:34.483051 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:34.483018    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2rq7_8f3f287d-baa3-45f9-9108-429e7bf73b9a/kube-rbac-proxy/0.log"
Sep 11 12:54:34.711653 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:34.711578    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mhljr_63bae1b4-810c-4092-a4ba-cc816e956adc/dns-node-resolver/0.log"
Sep 11 12:54:35.408969 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:35.408937    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6b29g_2dfc139d-214b-4e2f-8424-f97bc38788a0/node-ca/0.log"
Sep 11 12:54:36.740835 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:36.740800    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lb4vt_1446188b-058f-4d03-8064-059b91eedfe7/payload-pre-processing/0.log"
Sep 11 12:54:37.775329 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:37.775298    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbh52_2b5628a9-e957-4057-a70e-08d721e93717/serve-healthcheck-canary/0.log"
Sep 11 12:54:38.316825 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:38.316796    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2h56r_f56eb962-4c5c-4e68-a4c8-8cdf64e73d34/kube-rbac-proxy/0.log"
Sep 11 12:54:38.349016 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:38.348985    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2h56r_f56eb962-4c5c-4e68-a4c8-8cdf64e73d34/exporter/0.log"
Sep 11 12:54:38.380123 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:38.380096    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2h56r_f56eb962-4c5c-4e68-a4c8-8cdf64e73d34/extractor/0.log"
Sep 11 12:54:40.076663 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:40.076630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-r54n6"
Sep 11 12:54:40.672249 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:40.672216    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jhw8r_a7952582-3f47-43c5-99c9-c9f711169df2/manager/0.log"
Sep 11 12:54:41.204464 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:41.204360    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nwltg_7eba0dae-21aa-4501-9a7c-1ff4542a2609/manager/0.log"
Sep 11 12:54:41.298954 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:41.298921    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58467f79fd-mgksq_a47ee0d3-8455-48d3-987a-91040db6f3bc/manager/0.log"
Sep 11 12:54:41.449298 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:41.449266    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-k56gw_e3686c6d-9ad6-4114-ac21-c629251fd600/manager/0.log"
Sep 11 12:54:41.481975 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:41.481895    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-k56gw_e3686c6d-9ad6-4114-ac21-c629251fd600/copy-manifests/0.log"
Sep 11 12:54:42.911908 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:42.911822    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lb4vt_1446188b-058f-4d03-8064-059b91eedfe7/payload-pre-processing/0.log"
Sep 11 12:54:42.950493 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:42.950469    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69c967fbf8-mzdtf_74fa9b2d-13d8-4277-8509-5c25e3b2772b/manager/0.log"
Sep 11 12:54:43.002472 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:43.002437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vvcm8_e8b3ad9f-2ff2-43ab-976e-73288125a142/openshift-lws-operator/0.log"
Sep 11 12:54:44.857566 ip-10-0-135-5 kubenswrapper[2561]: I0911 12:54:44.857535    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lb4vt_1446188b-058f-4d03-8064-059b91eedfe7/payload-pre-processing/0.log"