Sep 12 02:36:21.594753 ip-10-0-128-128 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 02:36:22.026548 ip-10-0-128-128 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 02:36:22.026548 ip-10-0-128-128 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 02:36:22.026548 ip-10-0-128-128 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 02:36:22.026548 ip-10-0-128-128 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 02:36:22.026548 ip-10-0-128-128 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 02:36:22.027404 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.027316    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 02:36:22.032265 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032247    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032266    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032270    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032274    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032276    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032279    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032282    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032285    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032287    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032290    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032292    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032295    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032298    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032300    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032303    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032305    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032308    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032311    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:36:22.032305 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032314    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032316    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032319    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032321    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032324    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032326    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032329    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032331    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032333    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032336    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032338    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032344    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032346    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032348    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032351    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032353    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032355    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032358    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032361    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:36:22.032747 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032363    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032365    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032367    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032370    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032372    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032374    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032377    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032379    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032381    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032384    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032386    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032388    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032391    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032393    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032395    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032399    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032403    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032407    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032410    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:36:22.033216 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032413    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032417    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032421    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032423    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032426    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032429    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032432    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032435    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032437    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032440    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032442    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032459    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032462    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032464    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032467    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032469    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032473    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032476    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032478    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032481    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:36:22.033703 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032484    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032487    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032489    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032492    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032495    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032497    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032500    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032502    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032505    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032508    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032510    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032513    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032516    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032519    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032521    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032523    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032526    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032528    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032531    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032533    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:36:22.034183 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032536    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032538    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.032541    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033218    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033225    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033227    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033230    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033233    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033235    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033238    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033241    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033243    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033246    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033250    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033252    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033257    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033261    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033266    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033270    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033273    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:36:22.034693 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033276    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033279    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033281    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033284    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033287    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033290    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033292    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033295    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033297    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033300    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033302    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033305    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033308    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033310    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033313    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033315    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033318    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033320    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033323    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033325    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:36:22.035173 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033328    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033330    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033333    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033335    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033338    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033340    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033343    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033345    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033348    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033351    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033354    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033357    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033360    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033362    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033365    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033367    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033370    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033372    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033375    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:36:22.035663 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033377    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033380    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033382    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033385    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033388    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033390    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033394    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033398    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033401    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033404    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033406    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033409    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033411    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033414    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033416    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033418    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033421    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033423    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033426    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:36:22.036145 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033429    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033431    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033437    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033440    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033442    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033459    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033462    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033467    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033470    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033474    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033477    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033480    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033482    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033485    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033488    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033490    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033493    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033495    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033498    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:36:22.036643 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033501    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033504    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033507    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033509    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033512    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033878    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033883    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033885    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033888    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033891    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033893    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033896    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033899    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033902    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033904    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033909    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033911    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033913    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033919    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033922    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:36:22.037087 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033924    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033927    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033930    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033932    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033935    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033937    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033940    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033942    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033945    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033948    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033951    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033953    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033957    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033959    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033962    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033965    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033968    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033971    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033973    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033976    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033978    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:36:22.037611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033982    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033984    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033987    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033989    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033993    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.033997    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034001    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034004    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034007    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034010    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034014    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034018    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034021    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034024    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034026    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034029    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034032    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034034    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034037    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:36:22.038156 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034039    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034041    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034044    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034046    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034049    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034051    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034053    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034056    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034058    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034061    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034063    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034066    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034068    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034071    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034073    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034076    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034078    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034082    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034084    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034088    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:36:22.038710 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034090    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034093    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034095    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034097    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034100    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034103    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034105    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034108    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034110    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034113    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034115    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034118    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034120    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034122    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034125    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034128    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034130    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034133    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034135    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034138    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:36:22.039181 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034140    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034143    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034145    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034148    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034217    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034251    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034258    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034263    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034282    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034285    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034289    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034293    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034296    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034299    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034302    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034305    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034308    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034311    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034314    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034317    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034319    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034322    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034326    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 12 02:36:22.039717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034328    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034331    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034334    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034337    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034340    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034343    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034346    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034349    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034351    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034354    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034357    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034359    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034362    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034366    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034369    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034372    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034374    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034377    2579 flags.go:64] FLAG: --enable-server="true"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034379    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034384    2579 flags.go:64] FLAG: --event-burst="100"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034387    2579 flags.go:64] FLAG: --event-qps="50"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034390    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034393    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034395    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 12 02:36:22.040252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034399    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034401    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034404    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034408    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034411    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034414    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034416    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034419    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034422    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034425    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034428    2579 flags.go:64] FLAG: --feature-gates=""
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034431    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034434    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034437    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034440    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034443    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034461    2579 flags.go:64] FLAG: --help="false"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034464    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034466    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034469    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034472    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034475    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034479    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034482    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 02:36:22.040836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034484    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034487    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034490    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034492    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034496    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034516    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034520    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034523    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034527    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034530    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034533    2579 flags.go:64] FLAG: --lock-file=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034536    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034539    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034542    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034547    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034550    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034553    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034555    2579 flags.go:64] FLAG: --logging-format="text"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034558    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034561    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034564    2579 flags.go:64] FLAG: --manifest-url=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034567    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034573    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034576    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034580    2579 flags.go:64] FLAG: --max-pods="110"
Sep 12 02:36:22.041468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034582    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034585    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034588    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034591    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034594    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034597    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034600    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034606    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034609    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034612    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034615    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034618    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034622    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034625    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034628    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034631    2579 flags.go:64] FLAG: --port="10250"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034634    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034637    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c090ffddd8ee03d3"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034639    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034643    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034646    2579 flags.go:64] FLAG: --register-node="true"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034649    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034652    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034655    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 12 02:36:22.042139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034658    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034660    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034664    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034666    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034669    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034673    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034676    2579 flags.go:64] FLAG: --runonce="false"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034679    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034681    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034684    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034687    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034690    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034692    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034695    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034698    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034701    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034704    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034707    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034709    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034712    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034716    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034719    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034723    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034726    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034729    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 02:36:22.042751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034732    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034735    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034738    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034740    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034744    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034747    2579 flags.go:64] FLAG: --v="2"
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034750    2579 flags.go:64] FLAG: --version="false"
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034754    2579 flags.go:64] FLAG: --vmodule=""
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034757    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.034760    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034867    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034870    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034874    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034877    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034880    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034883    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034886    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034889    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034891    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034894    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034897    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034899    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:36:22.043367 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034902    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034904    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034906    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034909    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034911    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034915    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034917    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034920    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034922    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034925    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034927    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034929    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034932    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034934    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034937    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034939    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034942    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034944    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034947    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034950    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:36:22.043904 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034952    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034955    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034958    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034960    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034963    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034965    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034969    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034973    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034976    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034979    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034982    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034984    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034987    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034989    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034992    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034995    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.034997    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035002    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035005    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:36:22.044385 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035008    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035011    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035014    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035016    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035019    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035021    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035024    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035026    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035029    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035032    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035035    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035037    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035040    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035042    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035045    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035049    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035051    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035054    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035056    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:36:22.044854 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035060    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035063    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035065    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035067    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035070    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035072    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035075    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035077    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035080    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035082    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035084    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035088    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035090    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035093    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035096    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035098    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035101    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035103    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035106    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035108    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:36:22.045336 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035111    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035113    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035116    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035119    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035121    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035123    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035126    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035128    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.035132    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.035924    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.043413    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.043427    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.043457    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.043466    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 02:36:22.045862 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043521    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043526    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043530    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043533    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043536    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043538    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043542    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043546    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043549    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043551    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043554    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043556    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043559    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043561    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043564    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043566    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043569    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043571    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043574    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:36:22.046241 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043576    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043578    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043581    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043584    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043586    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043589    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043592    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043595    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043597    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043599    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043602    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043606    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043610    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043613    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043616    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043618    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043622    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043624    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043627    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043646    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:36:22.046764 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043649    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043653    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043656    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043658    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043661    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043663    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043666    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043668    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043671    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043673    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043676    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043679    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043682    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043684    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043686    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043689    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043692    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043694    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043697    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043699    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:36:22.047235 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043701    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043704    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043708    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043710    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043713    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043715    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043717    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043720    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043722    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043724    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043727    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043729    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043731    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043734    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043736    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043738    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043741    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043743    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043746    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043748    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:36:22.047829 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043750    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043754    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043756    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043759    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043761    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043763    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043766    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043770    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043773    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043776    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043779    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043782    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043784    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043787    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043790    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043793    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043795    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043798    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043800    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:36:22.048312 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043802    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.043807    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043912    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043916    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043919    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043922    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043925    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043927    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043930    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043932    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043934    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043937    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043940    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043943    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 02:36:22.048785 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043946    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043949    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043951    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043954    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043956    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043958    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043961    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043963    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043966    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043968    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043971    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043974    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043976    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043978    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043981    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043984    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043986    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043988    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043991    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043993    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 02:36:22.049148 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043995    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.043998    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044000    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044003    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044005    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044007    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044010    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044012    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044014    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044017    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044019    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044021    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044024    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044026    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044028    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044031    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044034    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044036    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044038    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044040    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 02:36:22.049682 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044043    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044046    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044049    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044051    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044054    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044056    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044058    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044061    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044063    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044065    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044067    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044070    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044072    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044074    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044077    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044079    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044082    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044085    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044087    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044089    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 02:36:22.050185 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044092    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044094    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044097    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044099    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044101    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044104    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044106    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044108    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044110    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044113    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044115    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044117    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044120    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044122    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044125    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044128    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044130    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044133    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044135    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 02:36:22.050799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044137    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044141    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044144    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044147    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044150    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044153    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044156    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:22.044159    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.044164    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.044253    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.046490    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.047468    2579 server.go:1019] "Starting client certificate rotation"
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.047580    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 02:36:22.051234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.047630    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 02:36:22.051631 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.050314    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 02:36:22.054861 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.054842    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 02:36:22.064489 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.064472    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 12 02:36:22.068383 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.067899    2579 log.go:25] "Validated CRI v1 image API"
Sep 12 02:36:22.069394 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.069371    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 02:36:22.072139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.072122    2579 fs.go:135] Filesystem UUIDs: map[1149f83f-311e-407a-a077-d5827f137cb6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7d5ba380-f648-42ae-906f-dfaf3bba32d4:/dev/nvme0n1p3]
Sep 12 02:36:22.072193 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.072139    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 02:36:22.077797 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.077684    2579 manager.go:216] Machine: {Timestamp:2026-09-12 02:36:22.076432434 +0000 UTC m=+0.359346004 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117255 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2182999d48c96ca3106102ae9a29de SystemUUID:ec218299-9d48-c96c-a310-6102ae9a29de BootID:43dc66c1-893c-476e-9737-34ee89851642 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:20:79:4b:60:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:20:79:4b:60:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:98:85:7b:f4:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 02:36:22.077797 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.077787    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 02:36:22.077929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.077866    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 02:36:22.079743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.079713    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 02:36:22.079882 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.079745    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 02:36:22.081842 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.081824    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 02:36:22.082467 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.082440    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 02:36:22.082467 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.082468    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 02:36:22.082534 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.082476    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 02:36:22.083412 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.083403    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 02:36:22.084387 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.084378    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 02:36:22.087569 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087558    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 02:36:22.087682 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087674    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 02:36:22.087710 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087703    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 02:36:22.087741 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087720    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 02:36:22.087741 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087734    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 02:36:22.087822 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087742    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 12 02:36:22.087822 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.087750    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 02:36:22.088075 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.088063    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 02:36:22.088112 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.088082    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 02:36:22.090225 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.090202    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 02:36:22.092235 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.092219    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 02:36:22.092311 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.092256    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 02:36:22.093966 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.093953    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 02:36:22.094040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.093974    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 02:36:22.094040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.093985    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 02:36:22.094040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.093994    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 02:36:22.094040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094007    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 02:36:22.094040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094015    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 02:36:22.094040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094036    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 02:36:22.094233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094044    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 02:36:22.094233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094054    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 02:36:22.094233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094062    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 02:36:22.094233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094081    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 02:36:22.094233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094091    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 02:36:22.094775 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094764    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 02:36:22.094836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.094778    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 02:36:22.098676 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.098544    2579 server.go:1268] "Started kubelet"
Sep 12 02:36:22.098676 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.098664    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 02:36:22.098785 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.098621    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 02:36:22.098785 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.098704    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 02:36:22.099289 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.099269    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 02:36:22.099595 ip-10-0-128-128 systemd[1]: Started Kubernetes Kubelet.
Sep 12 02:36:22.099918 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.099818    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 02:36:22.099918 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.099818    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 02:36:22.100311 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.100274    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 02:36:22.100895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.100826    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 02:36:22.105350 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.105324    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4pz6"
Sep 12 02:36:22.105876 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.105841    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 02:36:22.108536 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.108519    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 02:36:22.108608 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.108539    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 02:36:22.108608 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.107621    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d472428b024448  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-12 02:36:22.098519112 +0000 UTC m=+0.381432684,LastTimestamp:2026-09-12 02:36:22.098519112 +0000 UTC m=+0.381432684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 12 02:36:22.108802 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.108643    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.109395 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109267    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 02:36:22.109395 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109286    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 02:36:22.109395 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109378    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 02:36:22.109586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109522    2579 factory.go:55] Registering systemd factory
Sep 12 02:36:22.109586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109540    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 12 02:36:22.109586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109579    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 02:36:22.109586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109589    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 02:36:22.109768 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109674    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4pz6"
Sep 12 02:36:22.109997 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109931    2579 factory.go:153] Registering CRI-O factory
Sep 12 02:36:22.109997 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109945    2579 factory.go:223] Registration of the crio container factory successfully
Sep 12 02:36:22.109997 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.109993    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 02:36:22.110139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.110011    2579 factory.go:103] Registering Raw factory
Sep 12 02:36:22.110139 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.110027    2579 manager.go:1196] Started watching for new ooms in manager
Sep 12 02:36:22.110486 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.110473    2579 manager.go:349] Starting recovery of all containers
Sep 12 02:36:22.117217 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.117180    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:22.118288 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.118260    2579 manager.go:354] Recovery completed
Sep 12 02:36:22.120066 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.120046    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-128.ec2.internal\" not found" node="ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.123195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.123184    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.125094 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125081    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.125138 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125114    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.125138 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125130    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.125722 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125709    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 02:36:22.125780 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125720    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 02:36:22.125780 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125730    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.125780 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.125739    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 02:36:22.127159 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.127148    2579 policy_none.go:49] "None policy: Start"
Sep 12 02:36:22.127159 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.127163    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 02:36:22.127238 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.127176    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 02:36:22.127849 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.127835    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.127927 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.127864    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.127927 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.127879    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.128516 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.128504    2579 policy_none.go:47] "Start"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169126    2579 manager.go:337] "Starting Device Plugin manager"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.169161    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169171    2579 server.go:87] "Starting device plugin registration server"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169520    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169532    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169706    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169786    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.169793    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.170799    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 02:36:22.176932 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.170840    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.237431 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.237389    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 02:36:22.238754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.238732    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 02:36:22.238869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.238775    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 02:36:22.238869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.238797    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 02:36:22.238869 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.238831    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 12 02:36:22.241048 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.241030    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:22.269865 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.269837    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.270714 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.270698    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.270789 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.270726    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.270789 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.270736    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.270789 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.270763    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.276379 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.276366    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.276419 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.276386    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-128.ec2.internal\": node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.289919 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.289851    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.339318 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.339292    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"]
Sep 12 02:36:22.339469 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.339360    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.340289 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.340275    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.340347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.340302    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.340347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.340311    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.341753 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.341741    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.341852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.341838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.341900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.341867    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.342473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.342443    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.342538 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.342483    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.342538 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.342507    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.342538 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.342516    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.342635 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.342487    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.342635 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.342567    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.344027 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.344013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.344082 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.344041    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 02:36:22.344739 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.344718    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 02:36:22.344823 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.344750    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 02:36:22.344823 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.344762    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 12 02:36:22.357992 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.357970    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-128.ec2.internal\" not found" node="ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.361808 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.361794    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-128.ec2.internal\" not found" node="ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.390926 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.390906    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.410536 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.410514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbf1edaded94c099af13c14680243403-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"bbf1edaded94c099af13c14680243403\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.410536 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.410537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbf1edaded94c099af13c14680243403-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"bbf1edaded94c099af13c14680243403\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.410643 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.410551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b29c01eae268a79203b6e1217a43813-config\") pod \"kube-apiserver-proxy-ip-10-0-128-128.ec2.internal\" (UID: \"0b29c01eae268a79203b6e1217a43813\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.491694 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.491644    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.510998 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.510979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbf1edaded94c099af13c14680243403-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"bbf1edaded94c099af13c14680243403\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.511068 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.511053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbf1edaded94c099af13c14680243403-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"bbf1edaded94c099af13c14680243403\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.511107 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.511082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbf1edaded94c099af13c14680243403-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"bbf1edaded94c099af13c14680243403\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.511107 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.511005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbf1edaded94c099af13c14680243403-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"bbf1edaded94c099af13c14680243403\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.511171 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.511122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b29c01eae268a79203b6e1217a43813-config\") pod \"kube-apiserver-proxy-ip-10-0-128-128.ec2.internal\" (UID: \"0b29c01eae268a79203b6e1217a43813\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.511171 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.511147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b29c01eae268a79203b6e1217a43813-config\") pod \"kube-apiserver-proxy-ip-10-0-128-128.ec2.internal\" (UID: \"0b29c01eae268a79203b6e1217a43813\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.592504 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.592413    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.659852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.659814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.665767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:22.665417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:22.692902 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.692877    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.793478 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.793418    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.893945 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.893874    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:22.994544 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:22.994502    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 12 02:36:23.046961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.046933    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 02:36:23.047367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.047061    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 02:36:23.047367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.047091    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 02:36:23.076218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.076196    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:23.088411 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.088392    2579 apiserver.go:52] "Watching apiserver"
Sep 12 02:36:23.099149 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.099118    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 02:36:23.100201 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.100180    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-cggzr","openshift-dns/node-resolver-kf2vn","openshift-image-registry/node-ca-jrpjs","openshift-multus/multus-xn4b5","openshift-multus/network-metrics-daemon-nj5d4","openshift-network-diagnostics/network-check-target-pskg8","openshift-ovn-kubernetes/ovnkube-node-pvlnt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p","openshift-cluster-node-tuning-operator/tuned-hzj8g","openshift-multus/multus-additional-cni-plugins-nczqn","openshift-network-operator/iptables-alerter-k64l5","kube-system/global-pull-secret-syncer-jwttl"]
Sep 12 02:36:23.101911 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.101890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.103044 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.103026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.104433 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.104418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dz99t\""
Sep 12 02:36:23.104537 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.104418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 02:36:23.104537 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.104508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 02:36:23.104749 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.104723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.105342 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.105324    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p2rbf\""
Sep 12 02:36:23.105427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.105326    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.105427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.105401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.105998 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.105982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.107231 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.107212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.107314 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.107276    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:23.107563 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.107547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.107644 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.107545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 02:36:23.107749 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.107733    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zv8jm\""
Sep 12 02:36:23.107825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.107803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.108646 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:23.108734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108662    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 02:36:23.108734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108679    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.108734 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.108693    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:23.108734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108685    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.108734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108730    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s4stj\""
Sep 12 02:36:23.108955 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108821    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 02:36:23.108955 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108844    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:23.109051 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.108959    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 02:36:23.110556 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.110538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.113273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.112341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.113273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29tld\""
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113883    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113926    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 02:31:23 +0000 UTC" deadline="2028-06-21 08:39:10.161950052 +0000 UTC"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.113989    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15558h2m47.047965131s"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-cni-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-socket-dir-parent\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-netns\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-cni-bin\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-conf-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114213    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-run-netns\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqx7\" (UniqueName: \"kubernetes.io/projected/4d274728-2cb0-4d20-aee9-7841c219f3be-kube-api-access-zgqx7\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef2c7eeb-c2d2-40cb-a9ad-068694595411-hosts-file\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2c7eeb-c2d2-40cb-a9ad-068694595411-tmp-dir\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-log-socket\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tcrf\" (UniqueName: \"kubernetes.io/projected/ef2c7eeb-c2d2-40cb-a9ad-068694595411-kube-api-access-6tcrf\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.115261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-kubelet\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-ovnkube-script-lib\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-cnibin\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-hostroot\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ms5\" (UniqueName: \"kubernetes.io/projected/93a70706-fe88-4581-a774-f851f980d9b0-kube-api-access-l2ms5\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqpm\" (UniqueName: \"kubernetes.io/projected/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-kube-api-access-jnqpm\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-systemd-units\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-cni-bin\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-ovnkube-config\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-system-cni-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-kubelet\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pd64\" (UniqueName: \"kubernetes.io/projected/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-kube-api-access-4pd64\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-systemd\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-etc-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114963    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 02:36:23.116218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.114988    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.115028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.115130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-cni-multus\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.115940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.115993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93a70706-fe88-4581-a774-f851f980d9b0-multus-daemon-config\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-multus-certs\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-etc-kubernetes\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-host\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-serviceca\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d274728-2cb0-4d20-aee9-7841c219f3be-ovn-node-metrics-cert\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93a70706-fe88-4581-a774-f851f980d9b0-cni-binary-copy\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f2641236-d461-4b49-a03d-5f500d3fb573-agent-certs\") pod \"konnectivity-agent-cggzr\" (UID: \"f2641236-d461-4b49-a03d-5f500d3fb573\") " pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f2641236-d461-4b49-a03d-5f500d3fb573-konnectivity-ca\") pod \"konnectivity-agent-cggzr\" (UID: \"f2641236-d461-4b49-a03d-5f500d3fb573\") " pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-slash\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-var-lib-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-ovn\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-cni-netd\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-env-overrides\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116329    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.117743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-k8s-cni-cncf-io\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.117743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.116382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-node-log\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.117743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.117177    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 02:36:23.117743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.117265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.118241 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.118228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbmph\""
Sep 12 02:36:23.118620 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.118602    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.118748 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.118734    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.118825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.118812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.119220 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.119204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xvns8\""
Sep 12 02:36:23.119805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.119791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qtdnk\""
Sep 12 02:36:23.120079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.120064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 02:36:23.120378 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.120362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.120460 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.120423    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:23.120564 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.120526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 02:36:23.121339 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.121323    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 02:36:23.121412 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.121368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hthdg\""
Sep 12 02:36:23.121576 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.121562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 02:36:23.121642 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.121596    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 02:36:23.122035 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.122022    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"]
Sep 12 02:36:23.123025 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.123001    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 02:36:23.123112 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.123053    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 12 02:36:23.128087 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.128068    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 02:36:23.133533 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.133513    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 02:36:23.133693 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.133677    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"]
Sep 12 02:36:23.147403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.147388    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tfz29"
Sep 12 02:36:23.155829 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.155814    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tfz29"
Sep 12 02:36:23.210640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.210622    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 02:36:23.215855 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.215824    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b29c01eae268a79203b6e1217a43813.slice/crio-81709179cfd58a0e9f216fa42cc70b742b50785e1cd49a6793215963f59e9a6c WatchSource:0}: Error finding container 81709179cfd58a0e9f216fa42cc70b742b50785e1cd49a6793215963f59e9a6c: Status 404 returned error can't find the container with id 81709179cfd58a0e9f216fa42cc70b742b50785e1cd49a6793215963f59e9a6c
Sep 12 02:36:23.216157 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.216135    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf1edaded94c099af13c14680243403.slice/crio-fba0c02bd17c53de32380ae23c99e1937fa379d1194614fda83477811c6079b8 WatchSource:0}: Error finding container fba0c02bd17c53de32380ae23c99e1937fa379d1194614fda83477811c6079b8: Status 404 returned error can't find the container with id fba0c02bd17c53de32380ae23c99e1937fa379d1194614fda83477811c6079b8
Sep 12 02:36:23.217283 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-tmp\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.217366 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.217366 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jmh\" (UniqueName: \"kubernetes.io/projected/3af0f662-36c5-4666-a957-0ab576a736bc-kube-api-access-74jmh\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.217366 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tcrf\" (UniqueName: \"kubernetes.io/projected/ef2c7eeb-c2d2-40cb-a9ad-068694595411-kube-api-access-6tcrf\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.217544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.217544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-kubelet\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217544 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.217497    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:23.217544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-ovnkube-script-lib\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-kubelet\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.217552    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:36:23.717529779 +0000 UTC m=+2.000443339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g924z\" (UniqueName: \"kubernetes.io/projected/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-kube-api-access-g924z\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-cnibin\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ms5\" (UniqueName: \"kubernetes.io/projected/93a70706-fe88-4581-a774-f851f980d9b0-kube-api-access-l2ms5\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqpm\" (UniqueName: \"kubernetes.io/projected/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-kube-api-access-jnqpm\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-cnibin\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-cni-bin\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-ovnkube-config\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff9c9858-df11-4c55-bdb4-45f30f2d8502-iptables-alerter-script\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-device-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-systemd\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.217898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-cni-bin\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-socket-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.217988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-systemd\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-sys-fs\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqfk\" (UniqueName: \"kubernetes.io/projected/e34eea20-7fa0-475e-9d10-9d61cb4df890-kube-api-access-ztqfk\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-modprobe-d\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-kubernetes\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-host\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-cni-multus\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93a70706-fe88-4581-a774-f851f980d9b0-multus-daemon-config\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqx7\" (UniqueName: \"kubernetes.io/projected/4d274728-2cb0-4d20-aee9-7841c219f3be-kube-api-access-zgqx7\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-cni-multus\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-kubelet-config\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f2641236-d461-4b49-a03d-5f500d3fb573-agent-certs\") pod \"konnectivity-agent-cggzr\" (UID: \"f2641236-d461-4b49-a03d-5f500d3fb573\") " pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-var-lib-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.218730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-cni-netd\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-ovnkube-config\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-ovnkube-script-lib\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysctl-conf\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-var-lib-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-dbus\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-cni-netd\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysctl-d\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-sys\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-cni-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-netns\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-netns\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-cni-bin\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-run-netns\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-cni-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-cni-bin\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.219522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-run-netns\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.218800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xs6\" (UniqueName: \"kubernetes.io/projected/ff9c9858-df11-4c55-bdb4-45f30f2d8502-kube-api-access-49xs6\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-systemd\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93a70706-fe88-4581-a774-f851f980d9b0-multus-daemon-config\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-log-socket\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219559    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-log-socket\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-hostroot\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-systemd-units\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-system-cni-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-kubelet\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pd64\" (UniqueName: \"kubernetes.io/projected/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-kube-api-access-4pd64\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-systemd-units\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-etc-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.219986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-hostroot\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.220230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d274728-2cb0-4d20-aee9-7841c219f3be-ovn-node-metrics-cert\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-cnibin\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-multus-certs\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-etc-kubernetes\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-host\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-serviceca\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93a70706-fe88-4581-a774-f851f980d9b0-cni-binary-copy\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f2641236-d461-4b49-a03d-5f500d3fb573-konnectivity-ca\") pod \"konnectivity-agent-cggzr\" (UID: \"f2641236-d461-4b49-a03d-5f500d3fb573\") " pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-slash\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-etc-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-ovn\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-openvswitch\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-host\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.221030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.220497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-env-overrides\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff9c9858-df11-4c55-bdb4-45f30f2d8502-host-slash\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-k8s-cni-cncf-io\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-k8s-cni-cncf-io\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-system-cni-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-var-lib-kubelet\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-etc-kubernetes\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-host-run-multus-certs\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.221681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-run-ovn\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-node-log\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-registration-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysconfig\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-run\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-tuned\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-serviceca\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-socket-dir-parent\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-conf-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d274728-2cb0-4d20-aee9-7841c219f3be-env-overrides\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.221971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-system-cni-dir\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef2c7eeb-c2d2-40cb-a9ad-068694595411-hosts-file\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.222100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2c7eeb-c2d2-40cb-a9ad-068694595411-tmp-dir\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-lib-modules\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-var-lib-kubelet\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-host-slash\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93a70706-fe88-4581-a774-f851f980d9b0-cni-binary-copy\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-socket-dir-parent\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f2641236-d461-4b49-a03d-5f500d3fb573-konnectivity-ca\") pod \"konnectivity-agent-cggzr\" (UID: \"f2641236-d461-4b49-a03d-5f500d3fb573\") " pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.222973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93a70706-fe88-4581-a774-f851f980d9b0-multus-conf-dir\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.223072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d274728-2cb0-4d20-aee9-7841c219f3be-node-log\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.223115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.223103    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:36:23.223527 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.223280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef2c7eeb-c2d2-40cb-a9ad-068694595411-hosts-file\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.223527 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.223345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2c7eeb-c2d2-40cb-a9ad-068694595411-tmp-dir\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.228224 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.227252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqpm\" (UniqueName: \"kubernetes.io/projected/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-kube-api-access-jnqpm\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.228224 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.227317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d274728-2cb0-4d20-aee9-7841c219f3be-ovn-node-metrics-cert\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.228224 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.227613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f2641236-d461-4b49-a03d-5f500d3fb573-agent-certs\") pod \"konnectivity-agent-cggzr\" (UID: \"f2641236-d461-4b49-a03d-5f500d3fb573\") " pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.228224 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.227842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ms5\" (UniqueName: \"kubernetes.io/projected/93a70706-fe88-4581-a774-f851f980d9b0-kube-api-access-l2ms5\") pod \"multus-xn4b5\" (UID: \"93a70706-fe88-4581-a774-f851f980d9b0\") " pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.228562 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.228541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tcrf\" (UniqueName: \"kubernetes.io/projected/ef2c7eeb-c2d2-40cb-a9ad-068694595411-kube-api-access-6tcrf\") pod \"node-resolver-kf2vn\" (UID: \"ef2c7eeb-c2d2-40cb-a9ad-068694595411\") " pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.229029 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.229012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pd64\" (UniqueName: \"kubernetes.io/projected/1ed4de21-cc73-4384-8dd2-c4ed4710e18a-kube-api-access-4pd64\") pod \"node-ca-jrpjs\" (UID: \"1ed4de21-cc73-4384-8dd2-c4ed4710e18a\") " pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.229540 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.229474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqx7\" (UniqueName: \"kubernetes.io/projected/4d274728-2cb0-4d20-aee9-7841c219f3be-kube-api-access-zgqx7\") pod \"ovnkube-node-pvlnt\" (UID: \"4d274728-2cb0-4d20-aee9-7841c219f3be\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.230632 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.230616    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:23.230632 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.230634    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:23.230742 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.230644    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:23.230742 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.230738    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:23.730724553 +0000 UTC m=+2.013638111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:23.242172 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.242138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal" event={"ID":"0b29c01eae268a79203b6e1217a43813","Type":"ContainerStarted","Data":"81709179cfd58a0e9f216fa42cc70b742b50785e1cd49a6793215963f59e9a6c"}
Sep 12 02:36:23.243288 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.243271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" event={"ID":"bbf1edaded94c099af13c14680243403","Type":"ContainerStarted","Data":"fba0c02bd17c53de32380ae23c99e1937fa379d1194614fda83477811c6079b8"}
Sep 12 02:36:23.257639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.257620    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:23.322955 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.322924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-cnibin\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323067 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.322981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-cnibin\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323067 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff9c9858-df11-4c55-bdb4-45f30f2d8502-host-slash\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.323067 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323067 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-registration-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323067 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysconfig\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysconfig\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff9c9858-df11-4c55-bdb4-45f30f2d8502-host-slash\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-registration-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-run\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-tuned\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-system-cni-dir\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-lib-modules\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-run\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-var-lib-kubelet\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-tmp\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-system-cni-dir\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jmh\" (UniqueName: \"kubernetes.io/projected/3af0f662-36c5-4666-a957-0ab576a736bc-kube-api-access-74jmh\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-lib-modules\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g924z\" (UniqueName: \"kubernetes.io/projected/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-kube-api-access-g924z\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-var-lib-kubelet\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff9c9858-df11-4c55-bdb4-45f30f2d8502-iptables-alerter-script\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-device-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-socket-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-sys-fs\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqfk\" (UniqueName: \"kubernetes.io/projected/e34eea20-7fa0-475e-9d10-9d61cb4df890-kube-api-access-ztqfk\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-device-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-modprobe-d\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af0f662-36c5-4666-a957-0ab576a736bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-sys-fs\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-modprobe-d\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.323929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-kubernetes\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-host\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-socket-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-kubelet-config\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysctl-conf\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-kubernetes\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.323986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-dbus\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-kubelet-config\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysctl-d\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-sys\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-host\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xs6\" (UniqueName: \"kubernetes.io/projected/ff9c9858-df11-4c55-bdb4-45f30f2d8502-kube-api-access-49xs6\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-dbus\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.324755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-systemd\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-sys\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e34eea20-7fa0-475e-9d10-9d61cb4df890-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysctl-conf\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-sysctl-d\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-systemd\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.324321    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.324373    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:23.824355433 +0000 UTC m=+2.107268993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff9c9858-df11-4c55-bdb4-45f30f2d8502-iptables-alerter-script\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.325239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.324696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af0f662-36c5-4666-a957-0ab576a736bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.325874 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.325848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-etc-tuned\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.325961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.325886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-tmp\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.338227 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.338208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jmh\" (UniqueName: \"kubernetes.io/projected/3af0f662-36c5-4666-a957-0ab576a736bc-kube-api-access-74jmh\") pod \"multus-additional-cni-plugins-nczqn\" (UID: \"3af0f662-36c5-4666-a957-0ab576a736bc\") " pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.338494 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.338468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqfk\" (UniqueName: \"kubernetes.io/projected/e34eea20-7fa0-475e-9d10-9d61cb4df890-kube-api-access-ztqfk\") pod \"aws-ebs-csi-driver-node-hgx8p\" (UID: \"e34eea20-7fa0-475e-9d10-9d61cb4df890\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.338680 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.338664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g924z\" (UniqueName: \"kubernetes.io/projected/9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450-kube-api-access-g924z\") pod \"tuned-hzj8g\" (UID: \"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450\") " pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.339130 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.339108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xs6\" (UniqueName: \"kubernetes.io/projected/ff9c9858-df11-4c55-bdb4-45f30f2d8502-kube-api-access-49xs6\") pod \"iptables-alerter-k64l5\" (UID: \"ff9c9858-df11-4c55-bdb4-45f30f2d8502\") " pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.420585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.420526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:23.430239 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.430213    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2641236_d461_4b49_a03d_5f500d3fb573.slice/crio-d36496f3e79617094f2d4a21f75b320b2331d5076f60b07cf153f2dbc54fb4ed WatchSource:0}: Error finding container d36496f3e79617094f2d4a21f75b320b2331d5076f60b07cf153f2dbc54fb4ed: Status 404 returned error can't find the container with id d36496f3e79617094f2d4a21f75b320b2331d5076f60b07cf153f2dbc54fb4ed
Sep 12 02:36:23.438013 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.437995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kf2vn"
Sep 12 02:36:23.444671 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.444653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jrpjs"
Sep 12 02:36:23.445554 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.445509    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2c7eeb_c2d2_40cb_a9ad_068694595411.slice/crio-a207b97a421344531e5568ba48ed74385b8586dae7df019d56ce09c075f838c7 WatchSource:0}: Error finding container a207b97a421344531e5568ba48ed74385b8586dae7df019d56ce09c075f838c7: Status 404 returned error can't find the container with id a207b97a421344531e5568ba48ed74385b8586dae7df019d56ce09c075f838c7
Sep 12 02:36:23.452251 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.452229    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed4de21_cc73_4384_8dd2_c4ed4710e18a.slice/crio-c92a5550efa7c26f8626b3bc3432d9b15ac770b51076c99c861356e4058fdf87 WatchSource:0}: Error finding container c92a5550efa7c26f8626b3bc3432d9b15ac770b51076c99c861356e4058fdf87: Status 404 returned error can't find the container with id c92a5550efa7c26f8626b3bc3432d9b15ac770b51076c99c861356e4058fdf87
Sep 12 02:36:23.458455 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.458439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xn4b5"
Sep 12 02:36:23.464959 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.464942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:23.467397 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.467332    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a70706_fe88_4581_a774_f851f980d9b0.slice/crio-5393689132c6936c37ff9226b2c149a45f7985ab3cbe7533463b2201f6347a97 WatchSource:0}: Error finding container 5393689132c6936c37ff9226b2c149a45f7985ab3cbe7533463b2201f6347a97: Status 404 returned error can't find the container with id 5393689132c6936c37ff9226b2c149a45f7985ab3cbe7533463b2201f6347a97
Sep 12 02:36:23.473372 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.473352    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d274728_2cb0_4d20_aee9_7841c219f3be.slice/crio-9f39fa24c3f6882872b336b10ac97e7956f3e3846b28372e37ed0863bc79061a WatchSource:0}: Error finding container 9f39fa24c3f6882872b336b10ac97e7956f3e3846b28372e37ed0863bc79061a: Status 404 returned error can't find the container with id 9f39fa24c3f6882872b336b10ac97e7956f3e3846b28372e37ed0863bc79061a
Sep 12 02:36:23.476554 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.476537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p"
Sep 12 02:36:23.485507 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.485484    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34eea20_7fa0_475e_9d10_9d61cb4df890.slice/crio-49af52f6b93159c6dd7443012bcc2dec4ce52f703436af66395c048a7c631922 WatchSource:0}: Error finding container 49af52f6b93159c6dd7443012bcc2dec4ce52f703436af66395c048a7c631922: Status 404 returned error can't find the container with id 49af52f6b93159c6dd7443012bcc2dec4ce52f703436af66395c048a7c631922
Sep 12 02:36:23.492017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.491999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hzj8g"
Sep 12 02:36:23.499204 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.499187    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7fb3f9_ad2f_4743_9e0d_4f1ea4d1a450.slice/crio-0bd35010b99a273e0266b7691e3b6b463cf6339cf9f6f588ac7aeb845af69b98 WatchSource:0}: Error finding container 0bd35010b99a273e0266b7691e3b6b463cf6339cf9f6f588ac7aeb845af69b98: Status 404 returned error can't find the container with id 0bd35010b99a273e0266b7691e3b6b463cf6339cf9f6f588ac7aeb845af69b98
Sep 12 02:36:23.512425 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.512405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nczqn"
Sep 12 02:36:23.520646 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.520625    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af0f662_36c5_4666_a957_0ab576a736bc.slice/crio-a73c99769c2d598d2202a01e7b4b72c7a3dcec9e83307e70c1b6a6068f43e05e WatchSource:0}: Error finding container a73c99769c2d598d2202a01e7b4b72c7a3dcec9e83307e70c1b6a6068f43e05e: Status 404 returned error can't find the container with id a73c99769c2d598d2202a01e7b4b72c7a3dcec9e83307e70c1b6a6068f43e05e
Sep 12 02:36:23.532129 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.532105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k64l5"
Sep 12 02:36:23.540403 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:23.540384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9c9858_df11_4c55_bdb4_45f30f2d8502.slice/crio-c3a8628f80053a961b51ff9d9809ba02c7999d43523c31044d763505068d43bf WatchSource:0}: Error finding container c3a8628f80053a961b51ff9d9809ba02c7999d43523c31044d763505068d43bf: Status 404 returned error can't find the container with id c3a8628f80053a961b51ff9d9809ba02c7999d43523c31044d763505068d43bf
Sep 12 02:36:23.727425 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.727340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:23.727587 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.727498    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:23.727587 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.727562    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:36:24.727548155 +0000 UTC m=+3.010461718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:23.828121 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.828081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:23.828273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:23.828170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:23.828330 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.828275    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:23.828330 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.828284    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:23.828330 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.828301    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:23.828330 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.828315    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:23.828557 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.828346    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:24.828326971 +0000 UTC m=+3.111240554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:23.828557 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:23.828384    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:24.828353945 +0000 UTC m=+3.111267511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:24.139731 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.139595    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:24.158304 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.158265    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 02:31:24 +0000 UTC" deadline="2028-04-21 04:13:00.125151258 +0000 UTC"
Sep 12 02:36:24.158304 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.158303    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14089h36m35.966852756s"
Sep 12 02:36:24.275760 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.275713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k64l5" event={"ID":"ff9c9858-df11-4c55-bdb4-45f30f2d8502","Type":"ContainerStarted","Data":"c3a8628f80053a961b51ff9d9809ba02c7999d43523c31044d763505068d43bf"}
Sep 12 02:36:24.297848 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.297807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"9f39fa24c3f6882872b336b10ac97e7956f3e3846b28372e37ed0863bc79061a"}
Sep 12 02:36:24.322900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.322823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jrpjs" event={"ID":"1ed4de21-cc73-4384-8dd2-c4ed4710e18a","Type":"ContainerStarted","Data":"c92a5550efa7c26f8626b3bc3432d9b15ac770b51076c99c861356e4058fdf87"}
Sep 12 02:36:24.329878 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.329841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kf2vn" event={"ID":"ef2c7eeb-c2d2-40cb-a9ad-068694595411","Type":"ContainerStarted","Data":"a207b97a421344531e5568ba48ed74385b8586dae7df019d56ce09c075f838c7"}
Sep 12 02:36:24.347711 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.347678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cggzr" event={"ID":"f2641236-d461-4b49-a03d-5f500d3fb573","Type":"ContainerStarted","Data":"d36496f3e79617094f2d4a21f75b320b2331d5076f60b07cf153f2dbc54fb4ed"}
Sep 12 02:36:24.354009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.353921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerStarted","Data":"a73c99769c2d598d2202a01e7b4b72c7a3dcec9e83307e70c1b6a6068f43e05e"}
Sep 12 02:36:24.356869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.356847    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 02:36:24.372783 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.372747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hzj8g" event={"ID":"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450","Type":"ContainerStarted","Data":"0bd35010b99a273e0266b7691e3b6b463cf6339cf9f6f588ac7aeb845af69b98"}
Sep 12 02:36:24.388900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.388861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p" event={"ID":"e34eea20-7fa0-475e-9d10-9d61cb4df890","Type":"ContainerStarted","Data":"49af52f6b93159c6dd7443012bcc2dec4ce52f703436af66395c048a7c631922"}
Sep 12 02:36:24.404605 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.404420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xn4b5" event={"ID":"93a70706-fe88-4581-a774-f851f980d9b0","Type":"ContainerStarted","Data":"5393689132c6936c37ff9226b2c149a45f7985ab3cbe7533463b2201f6347a97"}
Sep 12 02:36:24.735893 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.735761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:24.736075 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.735992    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:24.736148 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.736097    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:36:26.736071738 +0000 UTC m=+5.018985319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:24.836846 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.836806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:24.837067 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:24.836965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:24.837130 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.837088    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:24.837130 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.837092    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:24.837130 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.837121    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:24.837264 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.837136    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:24.837264 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.837153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:26.837132058 +0000 UTC m=+5.120045637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:24.837264 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:24.837195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:26.837173573 +0000 UTC m=+5.120087142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:25.159194 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:25.159035    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 02:31:24 +0000 UTC" deadline="2028-06-20 05:31:01.214260976 +0000 UTC"
Sep 12 02:36:25.159194 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:25.159099    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15530h54m36.055166501s"
Sep 12 02:36:25.239131 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:25.239091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:25.239330 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:25.239229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:25.239330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:25.239324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:25.239457 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:25.239423    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:25.239518 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:25.239504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:25.240303 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:25.240273    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:26.752857 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:26.752816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:26.753619 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.753007    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:26.753619 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.753074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:36:30.753052688 +0000 UTC m=+9.035966276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:26.853147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:26.853233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.853359    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.853422    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:30.853402378 +0000 UTC m=+9.136315968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.853881    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.853900    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.853916    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:26.854092 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:26.853964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:30.853948605 +0000 UTC m=+9.136862182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:27.239717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:27.239619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:27.239899 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:27.239756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:27.240215 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:27.240194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:27.240342 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:27.240298    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:27.240420 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:27.240352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:27.240507 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:27.240444    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:29.239398 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:29.239364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:29.239917 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:29.239518    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:29.239917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:29.239911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:29.240039 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:29.240020    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:29.240097 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:29.240072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:29.240167 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:29.240143    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:30.789271 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:30.789133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:30.789834 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.789276    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:30.789834 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.789356    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:36:38.78933723 +0000 UTC m=+17.072250806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:30.890006 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:30.889966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:30.890194 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:30.890037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:30.890267 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.890198    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:30.890267 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.890231    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:30.890267 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.890246    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:30.890417 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.890308    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:38.890288102 +0000 UTC m=+17.173201666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:30.890646 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.890623    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:30.890739 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:30.890677    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:38.890662942 +0000 UTC m=+17.173576501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:31.239658 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:31.239617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:31.239821 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:31.239670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:31.239821 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:31.239629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:31.239821 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:31.239764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:31.239971 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:31.239895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:31.240008 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:31.239977    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:33.239923 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:33.239864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:33.240430 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:33.240019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:33.240430 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:33.239864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:33.240430 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:33.240101    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:33.240430 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:33.239896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:33.240430 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:33.240153    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:35.239433 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:35.239394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:35.239948 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:35.239444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:35.239948 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:35.239549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:35.239948 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:35.239568    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:35.239948 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:35.239634    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:35.239948 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:35.239702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:37.240099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:37.240035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:37.240099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:37.240063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:37.240099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:37.240071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:37.240664 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:37.240179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:37.240664 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:37.240219    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:37.240664 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:37.240299    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:38.852402 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:38.852370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:38.852929 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.852491    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:38.852929 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.852550    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:36:54.852532426 +0000 UTC m=+33.135445985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:38.953287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:38.953248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:38.953287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:38.953305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:38.953582 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.953395    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:38.953582 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.953458    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:38.953582 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.953473    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:38.953582 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.953478    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:54.953461655 +0000 UTC m=+33.236375240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:38.953582 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.953484    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:38.953582 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:38.953541    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:54.953519211 +0000 UTC m=+33.236432776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:39.239112 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:39.239084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:39.239245 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:39.239086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:39.239245 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:39.239171    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:39.239335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:39.239086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:39.239335 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:39.239254    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:39.239335 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:39.239316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:41.239815 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:41.239785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:41.240393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:41.239785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:41.240393 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:41.239896    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:41.240393 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:41.239977    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:41.240393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:41.239786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:41.240393 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:41.240055    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:43.239625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:43.239592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:43.239992 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:43.239592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:43.239992 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:43.239696    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:43.239992 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:43.239758    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:43.239992 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:43.239604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:43.239992 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:43.239826    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:44.451926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.451717    2579 generic.go:356] "Generic (PLEG): container finished" podID="bbf1edaded94c099af13c14680243403" containerID="064f086f11605008aa1c84c66d983d13f36f5e80798f3ebd495a31b5dcbc5b5f" exitCode=0
Sep 12 02:36:44.452513 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.451803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" event={"ID":"bbf1edaded94c099af13c14680243403","Type":"ContainerDied","Data":"064f086f11605008aa1c84c66d983d13f36f5e80798f3ebd495a31b5dcbc5b5f"}
Sep 12 02:36:44.453554 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.453531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal" event={"ID":"0b29c01eae268a79203b6e1217a43813","Type":"ContainerStarted","Data":"3c979235c5ce87d130d19080e2699c87c76659215012b34e7b0980d4718f0ea7"}
Sep 12 02:36:44.456135 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:36:44.456476 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456439    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d274728-2cb0-4d20-aee9-7841c219f3be" containerID="3bc38c2db0fc7e69a536cbd4530f8afadefd75c437c52a6109b8d4eb746b041c" exitCode=1
Sep 12 02:36:44.456476 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"c238ec5d82f7fb6da2cc5bc229a321e5d1f31fe7d0b6cdf00ca4558d23b15fcb"}
Sep 12 02:36:44.456640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"332dadf79fed81ecd7fcff13d3ff2a1f7094d8728093d5b66f7b5154af96ab6a"}
Sep 12 02:36:44.456640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"75368f21935583a281f728038c3c55bb40300a948069c37760f02593dde8d974"}
Sep 12 02:36:44.456640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"bdf86f5f4e96253620839ae0c8f37e9cd23803919ed6a4bac44968d18282a18f"}
Sep 12 02:36:44.456640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerDied","Data":"3bc38c2db0fc7e69a536cbd4530f8afadefd75c437c52a6109b8d4eb746b041c"}
Sep 12 02:36:44.456793 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.456656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"5d54044c10f4ad46bd892fe4e761a4789429954ba7bb3ba104da6b1b222f7563"}
Sep 12 02:36:44.457894 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.457873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jrpjs" event={"ID":"1ed4de21-cc73-4384-8dd2-c4ed4710e18a","Type":"ContainerStarted","Data":"68d31c67c38f5ded789e395b4bfbe100aba8e3ecbecc133091365afc1b549f00"}
Sep 12 02:36:44.459214 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.459197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kf2vn" event={"ID":"ef2c7eeb-c2d2-40cb-a9ad-068694595411","Type":"ContainerStarted","Data":"571f720687c376b5d5fa66535dfb548f003c7c5b1780e2559b4c90e01c2b33dc"}
Sep 12 02:36:44.460477 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.460437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cggzr" event={"ID":"f2641236-d461-4b49-a03d-5f500d3fb573","Type":"ContainerStarted","Data":"11aaab9ec094e6ae4864d20ecc3ac7b8fa08a4b4bb2fd52547a5b14c1ca5e68a"}
Sep 12 02:36:44.461859 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.461840    2579 generic.go:356] "Generic (PLEG): container finished" podID="3af0f662-36c5-4666-a957-0ab576a736bc" containerID="54c7a6d1139efaa8ea550fd59b1d475ac6fc16d717639bb6479293ed1ce00e47" exitCode=0
Sep 12 02:36:44.461932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.461917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerDied","Data":"54c7a6d1139efaa8ea550fd59b1d475ac6fc16d717639bb6479293ed1ce00e47"}
Sep 12 02:36:44.463317 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.463298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hzj8g" event={"ID":"9e7fb3f9-ad2f-4743-9e0d-4f1ea4d1a450","Type":"ContainerStarted","Data":"1e839f66ba3ac2f98f5b6a27f3b38674711575935e945dc98faec817d8870447"}
Sep 12 02:36:44.464809 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.464792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p" event={"ID":"e34eea20-7fa0-475e-9d10-9d61cb4df890","Type":"ContainerStarted","Data":"7192286c65c7042c57630716d8e697cb0d06e33678ffdb200c4f2dd5de2d8a80"}
Sep 12 02:36:44.466294 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.466277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xn4b5" event={"ID":"93a70706-fe88-4581-a774-f851f980d9b0","Type":"ContainerStarted","Data":"8db89dad2e7c372058f28aa6abf5ae143e3f769a1bd121a30c2e092aedccd696"}
Sep 12 02:36:44.478806 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.478764    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal" podStartSLOduration=20.478750763 podStartE2EDuration="20.478750763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:44.478178543 +0000 UTC m=+22.761092119" watchObservedRunningTime="2026-09-12 02:36:44.478750763 +0000 UTC m=+22.761664342"
Sep 12 02:36:44.492805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.492755    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cggzr" podStartSLOduration=1.587424005 podStartE2EDuration="21.492740062s" totalImagesPullingTime="19.905316057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:44.492649371 +0000 UTC m=+22.775562951" watchObservedRunningTime="2026-09-12 02:36:44.492740062 +0000 UTC m=+22.775653620"
Sep 12 02:36:44.505065 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.505030    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jrpjs" podStartSLOduration=1.6210667 podStartE2EDuration="21.505017295s" totalImagesPullingTime="19.883950595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:44.504829934 +0000 UTC m=+22.787743515" watchObservedRunningTime="2026-09-12 02:36:44.505017295 +0000 UTC m=+22.787930874"
Sep 12 02:36:44.542729 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.542683    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kf2vn" podStartSLOduration=1.652591819 podStartE2EDuration="21.542665071s" totalImagesPullingTime="19.890073252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:44.542252836 +0000 UTC m=+22.825166416" watchObservedRunningTime="2026-09-12 02:36:44.542665071 +0000 UTC m=+22.825578653"
Sep 12 02:36:44.558538 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.558486    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hzj8g" podStartSLOduration=1.7197029480000001 podStartE2EDuration="21.558472562s" totalImagesPullingTime="19.838769614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:44.55832086 +0000 UTC m=+22.841234440" watchObservedRunningTime="2026-09-12 02:36:44.558472562 +0000 UTC m=+22.841386139"
Sep 12 02:36:44.578629 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.578585    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xn4b5" podStartSLOduration=1.095986724 podStartE2EDuration="21.578550272s" totalImagesPullingTime="20.482563548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:44.57764284 +0000 UTC m=+22.860556420" watchObservedRunningTime="2026-09-12 02:36:44.578550272 +0000 UTC m=+22.861463853"
Sep 12 02:36:44.587039 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.587019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:44.694253 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.694223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:44.749168 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:44.749140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:45.239678 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.239650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:45.239906 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.239768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:45.239906 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:45.239780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:45.239906 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.239806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:45.239906 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:45.239894    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:45.240112 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:45.239977    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:45.470506 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.470465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k64l5" event={"ID":"ff9c9858-df11-4c55-bdb4-45f30f2d8502","Type":"ContainerStarted","Data":"0f098cd3ed98414b6c57ba30913cf35b1e368e2e4a26f77e4a03eaea2876260e"}
Sep 12 02:36:45.472596 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.472571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" event={"ID":"bbf1edaded94c099af13c14680243403","Type":"ContainerStarted","Data":"2d971730a73e68e0168a42320cf7be1d863b51cac054a8a5d5d69800c165ceb8"}
Sep 12 02:36:45.498885 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.498814    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k64l5" podStartSLOduration=2.701135924 podStartE2EDuration="22.498800298s" totalImagesPullingTime="19.797664374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:45.485417646 +0000 UTC m=+23.768331227" watchObservedRunningTime="2026-09-12 02:36:45.498800298 +0000 UTC m=+23.781713877"
Sep 12 02:36:45.499164 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.498931    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" podStartSLOduration=21.49892232 podStartE2EDuration="21.49892232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:45.498923833 +0000 UTC m=+23.781837395" watchObservedRunningTime="2026-09-12 02:36:45.49892232 +0000 UTC m=+23.781835900"
Sep 12 02:36:45.537490 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.537463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:45.610543 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.610385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:45.668579 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.668551    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 02:36:45.709620 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:45.709588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:46.180574 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.180471    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T02:36:45.668572875Z","UUID":"2db11071-4c39-4928-b2cc-a2b6289f418f","Handler":null,"Name":"","Endpoint":""}
Sep 12 02:36:46.182352 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.182325    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 02:36:46.182464 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.182360    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 02:36:46.478210 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.478137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:36:46.478636 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.478610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"ab9a63313aced82eeba459e8b29667de19f88de37ec5c672da41318e1905d92d"}
Sep 12 02:36:46.480513 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.480485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p" event={"ID":"e34eea20-7fa0-475e-9d10-9d61cb4df890","Type":"ContainerStarted","Data":"3b020162e7e069ef7b97100e338ef03e1291902147d965f415abf0fde3893c37"}
Sep 12 02:36:46.609891 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.609861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:46.660038 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:46.660007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:47.239468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.239420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:47.239642 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.239431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:47.239642 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:47.239552    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:47.239642 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.239560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:47.239642 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:47.239624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:47.239849 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:47.239719    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:47.395788 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.395754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:47.441586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.441561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:47.442585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.442562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:47.484957 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.484923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p" event={"ID":"e34eea20-7fa0-475e-9d10-9d61cb4df890","Type":"ContainerStarted","Data":"a53ec679343f9f4f0b8e9364769749d9366e9a07324e0006a522a859319913ca"}
Sep 12 02:36:47.485821 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.485801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cggzr"
Sep 12 02:36:47.503628 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.503547    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgx8p" podStartSLOduration=2.194039041 podStartE2EDuration="24.503531704s" totalImagesPullingTime="22.309492663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:47.503298818 +0000 UTC m=+25.786212389" watchObservedRunningTime="2026-09-12 02:36:47.503531704 +0000 UTC m=+25.786445286"
Sep 12 02:36:47.563884 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:47.563858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:48.728715 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:48.728686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:48.782657 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:48.782628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:49.239900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.239867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:49.240057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.239964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:49.240057 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:49.239968    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:49.240057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.240012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:49.240247 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:49.240101    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:49.240247 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:49.240200    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:49.492978 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.492902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:36:49.493314 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.493288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"361be403159e7058afb4301bd6739b4979e8dd53345efe3a4ec2c70a6bbf410d"}
Sep 12 02:36:49.493570 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.493552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:49.493782 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.493763    2579 scope.go:117] "RemoveContainer" containerID="3bc38c2db0fc7e69a536cbd4530f8afadefd75c437c52a6109b8d4eb746b041c"
Sep 12 02:36:49.495636 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.495615    2579 generic.go:356] "Generic (PLEG): container finished" podID="3af0f662-36c5-4666-a957-0ab576a736bc" containerID="176e04eeec2e58b07b4d3f97506411518a96431e02edd8ce7165f285e2fec9ba" exitCode=0
Sep 12 02:36:49.495735 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.495644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerDied","Data":"176e04eeec2e58b07b4d3f97506411518a96431e02edd8ce7165f285e2fec9ba"}
Sep 12 02:36:49.509395 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:49.509378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:50.502592 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.502498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:36:50.503543 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.502922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" event={"ID":"4d274728-2cb0-4d20-aee9-7841c219f3be","Type":"ContainerStarted","Data":"36e2f2fca9023f33a6d3f471be43f0797ef74f7cf8e67d9ffb2bbfb1cc0a3c2b"}
Sep 12 02:36:50.503792 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.503766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:50.503903 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.503891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:50.519661 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.519640    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:36:50.533681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.533627    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt" podStartSLOduration=7.581134113 podStartE2EDuration="27.533607497s" totalImagesPullingTime="19.952473384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:36:50.533076935 +0000 UTC m=+28.815990514" watchObservedRunningTime="2026-09-12 02:36:50.533607497 +0000 UTC m=+28.816521077"
Sep 12 02:36:50.862363 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.862178    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwttl"]
Sep 12 02:36:50.862546 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.862490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:50.862632 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:50.862611    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:50.865175 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.865147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pskg8"]
Sep 12 02:36:50.865313 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.865253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:50.865371 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:50.865349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:50.865752 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.865719    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nj5d4"]
Sep 12 02:36:50.865844 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:50.865820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:50.865913 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:50.865897    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:51.506766 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:51.506736    2579 generic.go:356] "Generic (PLEG): container finished" podID="3af0f662-36c5-4666-a957-0ab576a736bc" containerID="121b8158f57c41f55e42fdc0d4d48c0c79a008ac30e2af57656aa0200136514d" exitCode=0
Sep 12 02:36:51.507371 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:51.506816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerDied","Data":"121b8158f57c41f55e42fdc0d4d48c0c79a008ac30e2af57656aa0200136514d"}
Sep 12 02:36:52.239772 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:52.239743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:52.239947 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:52.239829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:52.239947 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:52.239875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:52.240032 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:52.239955    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:53.239603 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:53.239577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:53.239959 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:53.239678    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:53.513218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:53.513188    2579 generic.go:356] "Generic (PLEG): container finished" podID="3af0f662-36c5-4666-a957-0ab576a736bc" containerID="4e507794e5cc6b49ea3a2177cce11bf9a868c0cf48bc01c7b704657c62931a45" exitCode=0
Sep 12 02:36:53.513401 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:53.513239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerDied","Data":"4e507794e5cc6b49ea3a2177cce11bf9a868c0cf48bc01c7b704657c62931a45"}
Sep 12 02:36:54.239143 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:54.239105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:54.239324 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.239200    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:54.239324 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:54.239280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:54.239517 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.239352    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:54.881627 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:54.881588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:54.882021 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.881753    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:54.882021 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.881831    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:37:26.881809244 +0000 UTC m=+65.164722819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 02:36:54.982927 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:54.982890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:54.983089 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:54.982962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:54.983089 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.983047    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:54.983089 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.983056    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 02:36:54.983089 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.983075    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 02:36:54.983089 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.983087    2579 projected.go:196] Error preparing data for projected volume kube-api-access-s9q7j for pod openshift-network-diagnostics/network-check-target-pskg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:54.983358 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.983136    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j podName:b98d8787-a743-4a0a-9b94-6a6e093d34a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:26.983117993 +0000 UTC m=+65.266031566 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9q7j" (UniqueName: "kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j") pod "network-check-target-pskg8" (UID: "b98d8787-a743-4a0a-9b94-6a6e093d34a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 02:36:54.983358 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:54.983153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret podName:2a32de6d-b07f-4d06-ac75-8158fcaaa3a2 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:26.983144667 +0000 UTC m=+65.266058225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret") pod "global-pull-secret-syncer-jwttl" (UID: "2a32de6d-b07f-4d06-ac75-8158fcaaa3a2") : object "kube-system"/"original-pull-secret" not registered
Sep 12 02:36:55.239109 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:55.239024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:55.239264 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:55.239154    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:36:56.239195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.239159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:56.239677 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.239159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:56.239677 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:56.239262    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pskg8" podUID="b98d8787-a743-4a0a-9b94-6a6e093d34a2"
Sep 12 02:36:56.239677 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:56.239344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwttl" podUID="2a32de6d-b07f-4d06-ac75-8158fcaaa3a2"
Sep 12 02:36:56.461595 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.461567    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeReady"
Sep 12 02:36:56.461750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.461715    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 02:36:56.502265 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.502196    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pgkgx"]
Sep 12 02:36:56.518675 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.518429    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jskpx"]
Sep 12 02:36:56.518675 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.518559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.521630 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.521470    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 02:36:56.521630 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.521484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4279k\""
Sep 12 02:36:56.521630 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.521551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 02:36:56.530594 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.530567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgkgx"]
Sep 12 02:36:56.530692 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.530605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jskpx"]
Sep 12 02:36:56.530739 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.530698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.533673 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.533655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 02:36:56.533781 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.533709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 02:36:56.533989 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.533974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 02:36:56.534515 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.534498    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96kxp\""
Sep 12 02:36:56.612598 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.612565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:56.663461 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.663412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:36:56.698558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.698519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89pn\" (UniqueName: \"kubernetes.io/projected/08ba0220-aa22-4bbe-8901-8f2bf3d70320-kube-api-access-s89pn\") pod \"ingress-canary-jskpx\" (UID: \"08ba0220-aa22-4bbe-8901-8f2bf3d70320\") " pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.698725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.698583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-tmp-dir\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.698725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.698623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz48v\" (UniqueName: \"kubernetes.io/projected/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-kube-api-access-wz48v\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.698725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.698645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08ba0220-aa22-4bbe-8901-8f2bf3d70320-cert\") pod \"ingress-canary-jskpx\" (UID: \"08ba0220-aa22-4bbe-8901-8f2bf3d70320\") " pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.698725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.698692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-config-volume\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.698725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.698707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.799900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.799817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89pn\" (UniqueName: \"kubernetes.io/projected/08ba0220-aa22-4bbe-8901-8f2bf3d70320-kube-api-access-s89pn\") pod \"ingress-canary-jskpx\" (UID: \"08ba0220-aa22-4bbe-8901-8f2bf3d70320\") " pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.799900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.799867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-tmp-dir\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.800113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.799902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz48v\" (UniqueName: \"kubernetes.io/projected/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-kube-api-access-wz48v\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.800113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.799929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08ba0220-aa22-4bbe-8901-8f2bf3d70320-cert\") pod \"ingress-canary-jskpx\" (UID: \"08ba0220-aa22-4bbe-8901-8f2bf3d70320\") " pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.800113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.799967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-config-volume\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.800113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.799988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.800113 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:56.800075    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:56.800317 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:56.800126    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:57.300107884 +0000 UTC m=+35.583021458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:56.800317 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.800244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-tmp-dir\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.800759 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.800732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-config-volume\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.805020 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.804998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08ba0220-aa22-4bbe-8901-8f2bf3d70320-cert\") pod \"ingress-canary-jskpx\" (UID: \"08ba0220-aa22-4bbe-8901-8f2bf3d70320\") " pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.809059 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.808925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz48v\" (UniqueName: \"kubernetes.io/projected/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-kube-api-access-wz48v\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:56.809156 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.808924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89pn\" (UniqueName: \"kubernetes.io/projected/08ba0220-aa22-4bbe-8901-8f2bf3d70320-kube-api-access-s89pn\") pod \"ingress-canary-jskpx\" (UID: \"08ba0220-aa22-4bbe-8901-8f2bf3d70320\") " pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:56.839045 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:56.839024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jskpx"
Sep 12 02:36:57.240043 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:57.240008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:36:57.242932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:57.242907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9jkq\""
Sep 12 02:36:57.243066 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:57.242918    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 02:36:57.304767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:57.304737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:57.304960 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:57.304921    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:57.305025 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:57.305009    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:36:58.304981912 +0000 UTC m=+36.587895470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:58.093522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.093427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jskpx"]
Sep 12 02:36:58.098381 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:36:58.098299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ba0220_aa22_4bbe_8901_8f2bf3d70320.slice/crio-946e9e823c404a4c721f85eb172860ae156144790db1a8e1b264b2c0bcac25aa WatchSource:0}: Error finding container 946e9e823c404a4c721f85eb172860ae156144790db1a8e1b264b2c0bcac25aa: Status 404 returned error can't find the container with id 946e9e823c404a4c721f85eb172860ae156144790db1a8e1b264b2c0bcac25aa
Sep 12 02:36:58.239695 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.239668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:36:58.239852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.239832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:36:58.242724 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.242699    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 02:36:58.244007 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.243991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 02:36:58.244007 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.244000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 02:36:58.244163 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.243994    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7mt5x\""
Sep 12 02:36:58.312006 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.311981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:36:58.312111 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:58.312095    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:36:58.312181 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:36:58.312171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:00.312154583 +0000 UTC m=+38.595068142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:36:58.528512 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.528475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jskpx" event={"ID":"08ba0220-aa22-4bbe-8901-8f2bf3d70320","Type":"ContainerStarted","Data":"946e9e823c404a4c721f85eb172860ae156144790db1a8e1b264b2c0bcac25aa"}
Sep 12 02:36:58.533999 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.533969    2579 generic.go:356] "Generic (PLEG): container finished" podID="3af0f662-36c5-4666-a957-0ab576a736bc" containerID="2688862597ab8488d6e94c0be8250b745a93c9b95ae7a1838017082b8ec6007f" exitCode=0
Sep 12 02:36:58.534137 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:58.534013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerDied","Data":"2688862597ab8488d6e94c0be8250b745a93c9b95ae7a1838017082b8ec6007f"}
Sep 12 02:36:59.541054 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:59.541024    2579 generic.go:356] "Generic (PLEG): container finished" podID="3af0f662-36c5-4666-a957-0ab576a736bc" containerID="ad85083cca118cd8997e5a371d87cc76fda1a3e39ac90a794151d866c626407c" exitCode=0
Sep 12 02:36:59.541594 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:36:59.541069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerDied","Data":"ad85083cca118cd8997e5a371d87cc76fda1a3e39ac90a794151d866c626407c"}
Sep 12 02:37:00.328341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.328167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:37:00.328426 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:00.328307    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:37:00.328479 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:00.328436    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:04.328422309 +0000 UTC m=+42.611335868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:00.544824 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.544783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jskpx" event={"ID":"08ba0220-aa22-4bbe-8901-8f2bf3d70320","Type":"ContainerStarted","Data":"36418ddcd2d3cc7db57bd9974aeaa5c2cb85847fa78e410ab8a55ac28d100176"}
Sep 12 02:37:00.548143 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.548120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nczqn" event={"ID":"3af0f662-36c5-4666-a957-0ab576a736bc","Type":"ContainerStarted","Data":"f9379c79997ea238766abdeea5c37b8b2ee64b49dd920905ece91d55060aeb5a"}
Sep 12 02:37:00.559275 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.559205    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jskpx" podStartSLOduration=2.355196324 podStartE2EDuration="4.559189913s" totalImagesPullingTime="2.203993589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:00.558515975 +0000 UTC m=+38.841429557" watchObservedRunningTime="2026-09-12 02:37:00.559189913 +0000 UTC m=+38.842103491"
Sep 12 02:37:00.576822 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.576774    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nczqn" podStartSLOduration=6.443582462 podStartE2EDuration="37.576757101s" totalImagesPullingTime="31.133174639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:00.57631721 +0000 UTC m=+38.859230785" watchObservedRunningTime="2026-09-12 02:37:00.576757101 +0000 UTC m=+38.859670681"
Sep 12 02:37:00.695286 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.695256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:00.746495 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.746429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:00.827926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.827858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:00.929205 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:00.929174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:01.680585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:01.680557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:01.729827 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:01.729801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:04.354597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:04.354558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:37:04.354949 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:04.354704    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:37:04.354949 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:04.354776    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:12.354761109 +0000 UTC m=+50.637674667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:08.597496 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:08.597463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:08.648608 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:08.648576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:12.410171 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:12.410127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:37:12.410571 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:12.410279    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:37:12.410571 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:12.410344    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:37:28.410328439 +0000 UTC m=+66.693241997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:22.523355 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:22.523325    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvlnt"
Sep 12 02:37:22.630123 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:22.630092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:22.683910 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:22.683877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:37:24.447048 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.447020    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"]
Sep 12 02:37:24.449786 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.449769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.452319 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.452300    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 12 02:37:24.453635 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.453617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 12 02:37:24.453750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.453625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 12 02:37:24.453750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.453664    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 12 02:37:24.459228 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.459206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"]
Sep 12 02:37:24.465102 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.465075    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"]
Sep 12 02:37:24.467024 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.467009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.469617 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.469594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 12 02:37:24.469723 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.469627    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 12 02:37:24.469723 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.469659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 12 02:37:24.469723 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.469659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 12 02:37:24.477945 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.477924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"]
Sep 12 02:37:24.496649 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9657b2f1-2213-4f6d-814f-e0292cee6f9d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.496778 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-ca\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.496778 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56sdx\" (UniqueName: \"kubernetes.io/projected/4a96a6cb-d216-4c51-8cde-af1946a55be0-kube-api-access-56sdx\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.496778 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a96a6cb-d216-4c51-8cde-af1946a55be0-tmp\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.496778 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-hub\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.496778 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.496948 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.496948 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwhf\" (UniqueName: \"kubernetes.io/projected/9657b2f1-2213-4f6d-814f-e0292cee6f9d-kube-api-access-jjwhf\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.496948 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.496923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4a96a6cb-d216-4c51-8cde-af1946a55be0-klusterlet-config\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.597639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwhf\" (UniqueName: \"kubernetes.io/projected/9657b2f1-2213-4f6d-814f-e0292cee6f9d-kube-api-access-jjwhf\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.597777 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4a96a6cb-d216-4c51-8cde-af1946a55be0-klusterlet-config\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.597777 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9657b2f1-2213-4f6d-814f-e0292cee6f9d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.597777 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-ca\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.597777 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56sdx\" (UniqueName: \"kubernetes.io/projected/4a96a6cb-d216-4c51-8cde-af1946a55be0-kube-api-access-56sdx\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.597777 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a96a6cb-d216-4c51-8cde-af1946a55be0-tmp\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.598017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-hub\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.598017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.598017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.597853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.598203 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.598174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a96a6cb-d216-4c51-8cde-af1946a55be0-tmp\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.599089 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.599045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9657b2f1-2213-4f6d-814f-e0292cee6f9d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.602019 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.601989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-ca\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.602370 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.602347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4a96a6cb-d216-4c51-8cde-af1946a55be0-klusterlet-config\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.603965 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.603304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.605325 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.605293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-hub\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.606591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.606569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9657b2f1-2213-4f6d-814f-e0292cee6f9d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.606698 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.606602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56sdx\" (UniqueName: \"kubernetes.io/projected/4a96a6cb-d216-4c51-8cde-af1946a55be0-kube-api-access-56sdx\") pod \"klusterlet-addon-workmgr-66665cf88b-qmjb9\" (UID: \"4a96a6cb-d216-4c51-8cde-af1946a55be0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.606836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.606820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwhf\" (UniqueName: \"kubernetes.io/projected/9657b2f1-2213-4f6d-814f-e0292cee6f9d-kube-api-access-jjwhf\") pod \"cluster-proxy-proxy-agent-5668c584b8-jqbhw\" (UID: \"9657b2f1-2213-4f6d-814f-e0292cee6f9d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:24.759401 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.759308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:24.782613 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:24.782581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:37:25.028788 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:25.028716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"]
Sep 12 02:37:25.031601 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:37:25.031577    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a96a6cb_d216_4c51_8cde_af1946a55be0.slice/crio-1a8b58825113369c765b601ad8048f0fc02fa8b88f600e9dcea57915c2d22ad8 WatchSource:0}: Error finding container 1a8b58825113369c765b601ad8048f0fc02fa8b88f600e9dcea57915c2d22ad8: Status 404 returned error can't find the container with id 1a8b58825113369c765b601ad8048f0fc02fa8b88f600e9dcea57915c2d22ad8
Sep 12 02:37:25.051264 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:25.051239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"]
Sep 12 02:37:25.055675 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:37:25.055646    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9657b2f1_2213_4f6d_814f_e0292cee6f9d.slice/crio-f33fc92715add34ce87fccb00db4aa3c01d69200025d71c5b3715bb039e76e91 WatchSource:0}: Error finding container f33fc92715add34ce87fccb00db4aa3c01d69200025d71c5b3715bb039e76e91: Status 404 returned error can't find the container with id f33fc92715add34ce87fccb00db4aa3c01d69200025d71c5b3715bb039e76e91
Sep 12 02:37:25.609491 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:25.609441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9" event={"ID":"4a96a6cb-d216-4c51-8cde-af1946a55be0","Type":"ContainerStarted","Data":"1a8b58825113369c765b601ad8048f0fc02fa8b88f600e9dcea57915c2d22ad8"}
Sep 12 02:37:25.610439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:25.610418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" event={"ID":"9657b2f1-2213-4f6d-814f-e0292cee6f9d","Type":"ContainerStarted","Data":"f33fc92715add34ce87fccb00db4aa3c01d69200025d71c5b3715bb039e76e91"}
Sep 12 02:37:26.917013 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:26.916962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:37:26.919972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:26.919756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 02:37:26.928363 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:26.928078    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 02:37:26.928363 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:26.928151    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:38:30.928133024 +0000 UTC m=+129.211046586 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : secret "metrics-daemon-secret" not found
Sep 12 02:37:27.018069 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.018034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:37:27.018258 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.018109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:37:27.022314 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.022283    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 02:37:27.022591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.022565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 02:37:27.031002 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.030974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 02:37:27.034086 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.034065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a32de6d-b07f-4d06-ac75-8158fcaaa3a2-original-pull-secret\") pod \"global-pull-secret-syncer-jwttl\" (UID: \"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2\") " pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:37:27.048950 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.048867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9q7j\" (UniqueName: \"kubernetes.io/projected/b98d8787-a743-4a0a-9b94-6a6e093d34a2-kube-api-access-s9q7j\") pod \"network-check-target-pskg8\" (UID: \"b98d8787-a743-4a0a-9b94-6a6e093d34a2\") " pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:37:27.050273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.050217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwttl"
Sep 12 02:37:27.060802 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.060771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7mt5x\""
Sep 12 02:37:27.065626 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:27.065600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:37:28.429137 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:28.429104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:37:28.429625 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:28.429250    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:37:28.429625 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:37:28.429324    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:38:00.429304369 +0000 UTC m=+98.712217928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:37:31.546943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.546914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pskg8"]
Sep 12 02:37:31.547853 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.547831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwttl"]
Sep 12 02:37:31.550548 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:37:31.550508    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98d8787_a743_4a0a_9b94_6a6e093d34a2.slice/crio-c2a4e5dadd8dde79ab79b7dc1e3491cdc19fce154e18cd58f2ca5ef2b9fca2c8 WatchSource:0}: Error finding container c2a4e5dadd8dde79ab79b7dc1e3491cdc19fce154e18cd58f2ca5ef2b9fca2c8: Status 404 returned error can't find the container with id c2a4e5dadd8dde79ab79b7dc1e3491cdc19fce154e18cd58f2ca5ef2b9fca2c8
Sep 12 02:37:31.550988 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:37:31.550966    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a32de6d_b07f_4d06_ac75_8158fcaaa3a2.slice/crio-71f53251e0043fabfec14a8424e82ffbb28c87b2a8b8345179b1a36e2e38596b WatchSource:0}: Error finding container 71f53251e0043fabfec14a8424e82ffbb28c87b2a8b8345179b1a36e2e38596b: Status 404 returned error can't find the container with id 71f53251e0043fabfec14a8424e82ffbb28c87b2a8b8345179b1a36e2e38596b
Sep 12 02:37:31.630155 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.630121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9" event={"ID":"4a96a6cb-d216-4c51-8cde-af1946a55be0","Type":"ContainerStarted","Data":"d5b6859e632bf734f7d646d47c9a48735b2ba8d08e4a504726c424c36c9e7a45"}
Sep 12 02:37:31.630331 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.630316    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:31.631426 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.631400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pskg8" event={"ID":"b98d8787-a743-4a0a-9b94-6a6e093d34a2","Type":"ContainerStarted","Data":"c2a4e5dadd8dde79ab79b7dc1e3491cdc19fce154e18cd58f2ca5ef2b9fca2c8"}
Sep 12 02:37:31.632339 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.632318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:37:31.633768 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.633747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" event={"ID":"9657b2f1-2213-4f6d-814f-e0292cee6f9d","Type":"ContainerStarted","Data":"0526c05ce9087b714b9de814177db11a08a54772c650a898b3457adf164997ab"}
Sep 12 02:37:31.633863 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.633778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" event={"ID":"9657b2f1-2213-4f6d-814f-e0292cee6f9d","Type":"ContainerStarted","Data":"c062ffe7f05245b9c15afa181ac1c3b9a0d18242dfe9489885223e5db87c278f"}
Sep 12 02:37:31.633863 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.633793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" event={"ID":"9657b2f1-2213-4f6d-814f-e0292cee6f9d","Type":"ContainerStarted","Data":"5d9ed751169bfff464b8fc2abe50adb5c153a39d79c37e6a8d3f35bd294636c6"}
Sep 12 02:37:31.634927 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.634903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwttl" event={"ID":"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2","Type":"ContainerStarted","Data":"71f53251e0043fabfec14a8424e82ffbb28c87b2a8b8345179b1a36e2e38596b"}
Sep 12 02:37:31.645191 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.645135    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9" podStartSLOduration=1.399982391 podStartE2EDuration="7.645116551s" totalImagesPullingTime="6.24513416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:31.64431305 +0000 UTC m=+69.927226631" watchObservedRunningTime="2026-09-12 02:37:31.645116551 +0000 UTC m=+69.928030134"
Sep 12 02:37:31.675225 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:31.675180    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" podStartSLOduration=1.437499115 podStartE2EDuration="7.675164518s" totalImagesPullingTime="6.237665403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:31.674148036 +0000 UTC m=+69.957061617" watchObservedRunningTime="2026-09-12 02:37:31.675164518 +0000 UTC m=+69.958078099"
Sep 12 02:37:36.651588 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:36.651550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwttl" event={"ID":"2a32de6d-b07f-4d06-ac75-8158fcaaa3a2","Type":"ContainerStarted","Data":"2615424e10d660a61429b2f5bf2a7640f03131f41476f44974727d790bd3c7b5"}
Sep 12 02:37:36.653020 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:36.652994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pskg8" event={"ID":"b98d8787-a743-4a0a-9b94-6a6e093d34a2","Type":"ContainerStarted","Data":"4dc35167c2d1eb5f35b06ae8aee662d8163cbea0ffbc2de0c4f5b8af6a754838"}
Sep 12 02:37:36.653145 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:36.653127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:37:36.666625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:36.666582    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jwttl" podStartSLOduration=69.037123743 podStartE2EDuration="1m13.666568012s" totalImagesPullingTime="4.629444269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:36.665722548 +0000 UTC m=+74.948636128" watchObservedRunningTime="2026-09-12 02:37:36.666568012 +0000 UTC m=+74.949481626"
Sep 12 02:37:36.683277 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:37:36.683233    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pskg8" podStartSLOduration=69.05700335 podStartE2EDuration="1m13.683215921s" totalImagesPullingTime="4.626212571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:37:36.683042985 +0000 UTC m=+74.965956566" watchObservedRunningTime="2026-09-12 02:37:36.683215921 +0000 UTC m=+74.966129505"
Sep 12 02:38:00.451351 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:38:00.451312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:38:00.451846 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:38:00.451472    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:38:00.451846 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:38:00.451546    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:39:04.451527036 +0000 UTC m=+162.734440612 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:38:07.658870 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:38:07.658840    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pskg8"
Sep 12 02:38:30.958038 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:38:30.958000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:38:30.958442 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:38:30.958136    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 02:38:30.958442 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:38:30.958202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs podName:defb704d-5be6-4ab6-b3f0-e381e9dbb3aa nodeName:}" failed. No retries permitted until 2026-09-12 02:40:32.958181881 +0000 UTC m=+251.241095641 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs") pod "network-metrics-daemon-nj5d4" (UID: "defb704d-5be6-4ab6-b3f0-e381e9dbb3aa") : secret "metrics-daemon-secret" not found
Sep 12 02:38:59.530181 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:38:59.530134    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pgkgx" podUID="e80ade3b-dcdf-4c97-96ce-4d14057a7b64"
Sep 12 02:38:59.893218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:38:59.893188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:39:00.250591 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:39:00.250509    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nj5d4" podUID="defb704d-5be6-4ab6-b3f0-e381e9dbb3aa"
Sep 12 02:39:04.479285 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:04.479247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:39:04.479715 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:39:04.479392    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 02:39:04.479715 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:39:04.479472    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls podName:e80ade3b-dcdf-4c97-96ce-4d14057a7b64 nodeName:}" failed. No retries permitted until 2026-09-12 02:41:06.479440959 +0000 UTC m=+284.762354518 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls") pod "dns-default-pgkgx" (UID: "e80ade3b-dcdf-4c97-96ce-4d14057a7b64") : secret "dns-default-metrics-tls" not found
Sep 12 02:39:11.239713 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:11.239665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:39:31.630926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:31.630874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9" podUID="4a96a6cb-d216-4c51-8cde-af1946a55be0" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused"
Sep 12 02:39:31.982810 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:31.982777    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a96a6cb-d216-4c51-8cde-af1946a55be0" containerID="d5b6859e632bf734f7d646d47c9a48735b2ba8d08e4a504726c424c36c9e7a45" exitCode=1
Sep 12 02:39:31.982989 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:31.982825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9" event={"ID":"4a96a6cb-d216-4c51-8cde-af1946a55be0","Type":"ContainerDied","Data":"d5b6859e632bf734f7d646d47c9a48735b2ba8d08e4a504726c424c36c9e7a45"}
Sep 12 02:39:31.983202 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:31.983185    2579 scope.go:117] "RemoveContainer" containerID="d5b6859e632bf734f7d646d47c9a48735b2ba8d08e4a504726c424c36c9e7a45"
Sep 12 02:39:32.986971 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:32.986935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9" event={"ID":"4a96a6cb-d216-4c51-8cde-af1946a55be0","Type":"ContainerStarted","Data":"5156c4fcf93c7b724caad36f14e3c5011bd1487b9f5a948e67b9c9bd0b06adb3"}
Sep 12 02:39:32.987407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:32.987273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:39:32.987870 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:32.987854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66665cf88b-qmjb9"
Sep 12 02:39:40.690535 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:39:40.690501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:03.681079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:03.681047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:03.729793 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:03.729766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:06.162641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:06.162610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:06.264587 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:06.264555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:08.110932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.110899    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tdw5q"]
Sep 12 02:40:08.112923 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.112907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.115794 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.115776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 02:40:08.116707 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.116691    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 02:40:08.117260 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.117248    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-klkm9\""
Sep 12 02:40:08.118281 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.118256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 02:40:08.118818 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.118805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 02:40:08.131898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.131876    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:08.147855 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.147834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdw5q"]
Sep 12 02:40:08.192745 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.192717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.192745 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.192747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffdw\" (UniqueName: \"kubernetes.io/projected/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-api-access-pffdw\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.192921 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.192769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c285a525-1cfe-4e6a-9f24-072e1788a672-crio-socket\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.192921 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.192841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.192921 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.192903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c285a525-1cfe-4e6a-9f24-072e1788a672-data-volume\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.211037 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.211014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:08.293273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c285a525-1cfe-4e6a-9f24-072e1788a672-crio-socket\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.293407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.293407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c285a525-1cfe-4e6a-9f24-072e1788a672-data-volume\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.293407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c285a525-1cfe-4e6a-9f24-072e1788a672-crio-socket\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.293407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.293637 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffdw\" (UniqueName: \"kubernetes.io/projected/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-api-access-pffdw\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.293637 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.293494    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.293637 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.293537    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.293637 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.293593    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:08.793572296 +0000 UTC m=+227.076485869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.293637 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.293609    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:08.79360025 +0000 UTC m=+227.076513811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.293871 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.293693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c285a525-1cfe-4e6a-9f24-072e1788a672-data-volume\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.304504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.304479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffdw\" (UniqueName: \"kubernetes.io/projected/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-api-access-pffdw\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.568529 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.568419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:08.614612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.614589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:08.798115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.798065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.798282 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.798131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:08.798282 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.798196    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.798282 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.798219    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.798282 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.798256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:09.79824268 +0000 UTC m=+228.081156238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:08.798282 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:08.798280    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:09.798266588 +0000 UTC m=+228.081180147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:08.999972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:08.999943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.049297 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.049270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.273735 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.273666    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.372728 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.372697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.541285 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.541214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.637278 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.637244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.736195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.736169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:09.807559 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.807487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:09.807559 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:09.807547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:09.807733 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:09.807605    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:09.807733 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:09.807651    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:09.807733 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:09.807671    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:11.807655768 +0000 UTC m=+230.090569331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:09.807733 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:09.807689    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:11.807678327 +0000 UTC m=+230.090591886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:10.271383 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:10.271355    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kf2vn_ef2c7eeb-c2d2-40cb-a9ad-068694595411/dns-node-resolver/0.log"
Sep 12 02:40:10.906979 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:10.906947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:10.955442 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:10.955412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:11.073411 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:11.073387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jrpjs_1ed4de21-cc73-4384-8dd2-c4ed4710e18a/node-ca/0.log"
Sep 12 02:40:11.752852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:11.752822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:11.823021 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:11.822981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:11.823144 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:11.823045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:11.823144 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:11.823114    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:11.823144 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:11.823127    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:11.823264 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:11.823172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:15.823157913 +0000 UTC m=+234.106071471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:11.823264 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:11.823186    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:15.823180515 +0000 UTC m=+234.106094073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:11.873121 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:11.873093    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jskpx_08ba0220-aa22-4bbe-8901-8f2bf3d70320/serve-healthcheck-canary/0.log"
Sep 12 02:40:12.720645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:12.720611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:12.912681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:12.912650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:13.013381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:13.013303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:13.066099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:13.066076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:13.124591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:13.124561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:13.225677 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:13.225649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:14.342756 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:14.342697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:14.443602 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:14.443547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:14.616547 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:14.616518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:14.938231 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:14.938149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:15.041170 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:15.041140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:15.212671 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:15.212588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:15.617760 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:15.617731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:15.852933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:15.852893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:15.853133 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:15.852980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:15.853133 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:15.853009    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 02:40:15.853133 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:15.853088    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:23.853070758 +0000 UTC m=+242.135984316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : configmap "kube-rbac-proxy" not found
Sep 12 02:40:15.853133 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:15.853100    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:15.853298 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:40:15.853161    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls podName:c285a525-1cfe-4e6a-9f24-072e1788a672 nodeName:}" failed. No retries permitted until 2026-09-12 02:40:23.85314637 +0000 UTC m=+242.136059932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdw5q" (UID: "c285a525-1cfe-4e6a-9f24-072e1788a672") : secret "insights-runtime-extractor-tls" not found
Sep 12 02:40:16.212664 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:16.212634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:16.511362 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:16.511281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:16.617278 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:16.617249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:16.717244 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:16.717222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:17.012988 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:17.012960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:17.212850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:17.212820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:17.809235 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:17.809199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:18.272536 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:18.272503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:18.410893 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:18.410866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:18.610881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:18.610854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:19.010068 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:19.009991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:19.211023 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:19.210990    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:19.259974 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:19.259950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:19.610379 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:19.610349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:20.412935 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:20.412903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:20.615426 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:20.615398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:21.009526 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:21.009496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:21.211605 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:21.211551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:21.383050 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:21.383019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:21.807597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:21.807518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:22.418375 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:22.418344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:22.620921 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:22.620888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:23.011004 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.010956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:23.213399 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.213371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:23.414810 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.414785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:23.914922 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.914890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:23.915079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.914951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:23.915391 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.915367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c285a525-1cfe-4e6a-9f24-072e1788a672-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:23.917291 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:23.917271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c285a525-1cfe-4e6a-9f24-072e1788a672-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdw5q\" (UID: \"c285a525-1cfe-4e6a-9f24-072e1788a672\") " pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:24.023628 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:24.023595    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-klkm9\""
Sep 12 02:40:24.031895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:24.031875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdw5q"
Sep 12 02:40:24.298428 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:24.298353    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdw5q"]
Sep 12 02:40:24.302325 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:40:24.302301    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc285a525_1cfe_4e6a_9f24_072e1788a672.slice/crio-1adecb153307168a4ee5bf930cc2d2acb8883786d57d29a4ee78613e30f28dce WatchSource:0}: Error finding container 1adecb153307168a4ee5bf930cc2d2acb8883786d57d29a4ee78613e30f28dce: Status 404 returned error can't find the container with id 1adecb153307168a4ee5bf930cc2d2acb8883786d57d29a4ee78613e30f28dce
Sep 12 02:40:25.136438 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:25.136407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdw5q" event={"ID":"c285a525-1cfe-4e6a-9f24-072e1788a672","Type":"ContainerStarted","Data":"84f58ecad91ef8f1524b943f46fab83d2d13e4cd9fff7698da3ab0e35c1b8fec"}
Sep 12 02:40:25.136797 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:25.136492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdw5q" event={"ID":"c285a525-1cfe-4e6a-9f24-072e1788a672","Type":"ContainerStarted","Data":"f0401d751d839787f41564d36e34715aef579041e2528ed3810c1a418f99dc2d"}
Sep 12 02:40:25.136797 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:25.136520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdw5q" event={"ID":"c285a525-1cfe-4e6a-9f24-072e1788a672","Type":"ContainerStarted","Data":"1adecb153307168a4ee5bf930cc2d2acb8883786d57d29a4ee78613e30f28dce"}
Sep 12 02:40:27.144214 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:27.144181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdw5q" event={"ID":"c285a525-1cfe-4e6a-9f24-072e1788a672","Type":"ContainerStarted","Data":"958a8a0c7a67a18bf4e96a71eb8d02ca191d6c2232b5476f8ed3b4d56815336c"}
Sep 12 02:40:27.162247 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:27.162191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tdw5q" podStartSLOduration=17.164314241 podStartE2EDuration="19.1621762s" totalImagesPullingTime="1.997861959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:40:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:27.161393706 +0000 UTC m=+245.444307285" watchObservedRunningTime="2026-09-12 02:40:27.1621762 +0000 UTC m=+245.445089779"
Sep 12 02:40:31.316414 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.316384    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fdf8bffd8-6278h"]
Sep 12 02:40:31.318611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.318597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.321109 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.321085    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 12 02:40:31.322296 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.322281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 12 02:40:31.322395 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.322325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jtp6k\""
Sep 12 02:40:31.322500 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.322484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 12 02:40:31.326936 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.326917    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 12 02:40:31.332330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.332311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdf8bffd8-6278h"]
Sep 12 02:40:31.382587 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.382553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:31.466880 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.466853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-image-registry-private-configuration\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.466967 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.466884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-installation-pull-secrets\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.466967 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.466912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-trusted-ca\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.467085 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.467004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-registry-tls\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.467134 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.467096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-ca-trust-extracted\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.467134 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.467126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-bound-sa-token\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.467227 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.467149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj8r\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-kube-api-access-tzj8r\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.467227 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.467183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-registry-certificates\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.481936 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.481912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:31.567546 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-registry-tls\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567546 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-ca-trust-extracted\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567682 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-bound-sa-token\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567724 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj8r\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-kube-api-access-tzj8r\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567774 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-registry-certificates\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567774 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-image-registry-private-configuration\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567868 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-installation-pull-secrets\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567868 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-trusted-ca\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.567961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.567933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-ca-trust-extracted\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.568507 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.568489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-registry-certificates\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.569046 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.569022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-trusted-ca\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.570196 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.570174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-installation-pull-secrets\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.570278 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.570224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-registry-tls\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.570360 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.570301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-image-registry-private-configuration\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.575432 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.575415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-bound-sa-token\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.575840 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.575825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj8r\" (UniqueName: \"kubernetes.io/projected/cdd3e22f-3a82-4f78-9e5a-83ca725b240d-kube-api-access-tzj8r\") pod \"image-registry-6fdf8bffd8-6278h\" (UID: \"cdd3e22f-3a82-4f78-9e5a-83ca725b240d\") " pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.627658 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.627642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:31.898025 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:31.897993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdf8bffd8-6278h"]
Sep 12 02:40:31.901922 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:40:31.901895    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd3e22f_3a82_4f78_9e5a_83ca725b240d.slice/crio-b61444813d3add7c7b101f8f9db26e798663cad818f42adccb44e544a2a2fc0a WatchSource:0}: Error finding container b61444813d3add7c7b101f8f9db26e798663cad818f42adccb44e544a2a2fc0a: Status 404 returned error can't find the container with id b61444813d3add7c7b101f8f9db26e798663cad818f42adccb44e544a2a2fc0a
Sep 12 02:40:32.160815 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:32.160719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h" event={"ID":"cdd3e22f-3a82-4f78-9e5a-83ca725b240d","Type":"ContainerStarted","Data":"07585d366fc35fc04f7f9fbe1b2666995edac624c6b1d2dd75ea6fb23e991e56"}
Sep 12 02:40:32.160815 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:32.160771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h" event={"ID":"cdd3e22f-3a82-4f78-9e5a-83ca725b240d","Type":"ContainerStarted","Data":"b61444813d3add7c7b101f8f9db26e798663cad818f42adccb44e544a2a2fc0a"}
Sep 12 02:40:32.161036 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:32.160854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:40:32.181773 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:32.179028    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h" podStartSLOduration=1.179007713 podStartE2EDuration="1.179007713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:32.178035608 +0000 UTC m=+250.460949187" watchObservedRunningTime="2026-09-12 02:40:32.179007713 +0000 UTC m=+250.461921293"
Sep 12 02:40:32.979167 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:32.979133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:40:32.981540 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:32.981523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defb704d-5be6-4ab6-b3f0-e381e9dbb3aa-metrics-certs\") pod \"network-metrics-daemon-nj5d4\" (UID: \"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa\") " pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:40:33.143265 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:33.143234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9jkq\""
Sep 12 02:40:33.151330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:33.151315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nj5d4"
Sep 12 02:40:33.417496 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:33.417466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nj5d4"]
Sep 12 02:40:33.420368 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:40:33.420338    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefb704d_5be6_4ab6_b3f0_e381e9dbb3aa.slice/crio-3eb6ea027fd916b26248ba4717eb840ad3eb713630ffe02a900b3183d35eaf42 WatchSource:0}: Error finding container 3eb6ea027fd916b26248ba4717eb840ad3eb713630ffe02a900b3183d35eaf42: Status 404 returned error can't find the container with id 3eb6ea027fd916b26248ba4717eb840ad3eb713630ffe02a900b3183d35eaf42
Sep 12 02:40:33.724621 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:33.724534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:33.772993 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:33.772962    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:34.168978 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:34.168942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nj5d4" event={"ID":"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa","Type":"ContainerStarted","Data":"3eb6ea027fd916b26248ba4717eb840ad3eb713630ffe02a900b3183d35eaf42"}
Sep 12 02:40:35.173984 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:35.173951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nj5d4" event={"ID":"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa","Type":"ContainerStarted","Data":"35b49c3f28ba2a8e0f59b1b375c2af2622cecde815528af3a855693d87ed3246"}
Sep 12 02:40:35.174423 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:35.173998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nj5d4" event={"ID":"defb704d-5be6-4ab6-b3f0-e381e9dbb3aa","Type":"ContainerStarted","Data":"2982df070aeeef460ae377aa087ac8de55709e880ffd4e4e1215a8e15b473365"}
Sep 12 02:40:35.189746 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:35.189708    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nj5d4" podStartSLOduration=251.23934892 podStartE2EDuration="4m12.189694411s" totalImagesPullingTime="950.345491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:35.189620065 +0000 UTC m=+253.472533644" watchObservedRunningTime="2026-09-12 02:40:35.189694411 +0000 UTC m=+253.472607990"
Sep 12 02:40:35.255609 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:35.255582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:35.356486 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:35.356440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:35.457841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:35.457766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:37.893317 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:37.893288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:37.993530 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:37.993499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:38.062865 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:38.062836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:38.358969 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:38.358941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:41.075394 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:41.075351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:41.138023 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:41.137997    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:41.204712 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:41.204680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:43.024620 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:43.024548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.217344 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.217312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.377403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.377373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.452146 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.452112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.479872 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.479798    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7slnk"]
Sep 12 02:40:45.482378 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.482358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.486326 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gqzx9\""
Sep 12 02:40:45.486439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486362    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 02:40:45.486439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486365    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 02:40:45.486439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486429    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 02:40:45.486612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486365    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 02:40:45.486612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 02:40:45.486737 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.486724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 02:40:45.530898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.530867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.561121 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8b5n\" (UniqueName: \"kubernetes.io/projected/79c2dd42-3472-444a-8710-de01329994af-kube-api-access-x8b5n\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561226 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79c2dd42-3472-444a-8710-de01329994af-node-exporter-textfile\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561226 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-node-exporter-wtmp\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561226 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c2dd42-3472-444a-8710-de01329994af-node-exporter-tls\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-sys\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-root\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79c2dd42-3472-444a-8710-de01329994af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c2dd42-3472-444a-8710-de01329994af-metrics-client-ca\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.561335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.561323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79c2dd42-3472-444a-8710-de01329994af-node-exporter-accelerators-collector-config\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.629611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.629584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.661734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.661715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79c2dd42-3472-444a-8710-de01329994af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.661818 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.661741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c2dd42-3472-444a-8710-de01329994af-metrics-client-ca\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.661818 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.661763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79c2dd42-3472-444a-8710-de01329994af-node-exporter-accelerators-collector-config\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.661818 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.661792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8b5n\" (UniqueName: \"kubernetes.io/projected/79c2dd42-3472-444a-8710-de01329994af-kube-api-access-x8b5n\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.661950 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.661927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79c2dd42-3472-444a-8710-de01329994af-node-exporter-textfile\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.661997 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.661965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-node-exporter-wtmp\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662047 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c2dd42-3472-444a-8710-de01329994af-node-exporter-tls\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662132 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-sys\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662324 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-root\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-node-exporter-wtmp\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-sys\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79c2dd42-3472-444a-8710-de01329994af-node-exporter-textfile\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79c2dd42-3472-444a-8710-de01329994af-root\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c2dd42-3472-444a-8710-de01329994af-metrics-client-ca\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.662830 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.662469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79c2dd42-3472-444a-8710-de01329994af-node-exporter-accelerators-collector-config\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.664219 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.664203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79c2dd42-3472-444a-8710-de01329994af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.664309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.664290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79c2dd42-3472-444a-8710-de01329994af-node-exporter-tls\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.670606 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.670589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8b5n\" (UniqueName: \"kubernetes.io/projected/79c2dd42-3472-444a-8710-de01329994af-kube-api-access-x8b5n\") pod \"node-exporter-7slnk\" (UID: \"79c2dd42-3472-444a-8710-de01329994af\") " pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.729744 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.729720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:45.791635 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.791559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7slnk"
Sep 12 02:40:45.803430 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:40:45.803407    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c2dd42_3472_444a_8710_de01329994af.slice/crio-beb44ddb73499e39639d0e6deb91d77ab7892890b1cc5743caf912b910ac717a WatchSource:0}: Error finding container beb44ddb73499e39639d0e6deb91d77ab7892890b1cc5743caf912b910ac717a: Status 404 returned error can't find the container with id beb44ddb73499e39639d0e6deb91d77ab7892890b1cc5743caf912b910ac717a
Sep 12 02:40:45.815410 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:45.815383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:46.188319 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:46.188288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:46.209631 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:46.209597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7slnk" event={"ID":"79c2dd42-3472-444a-8710-de01329994af","Type":"ContainerStarted","Data":"beb44ddb73499e39639d0e6deb91d77ab7892890b1cc5743caf912b910ac717a"}
Sep 12 02:40:46.437712 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:46.437680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:46.512608 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:46.512526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:47.035660 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:47.035578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:47.214374 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:47.214345    2579 generic.go:356] "Generic (PLEG): container finished" podID="79c2dd42-3472-444a-8710-de01329994af" containerID="3e556d2f7e3e2b0f9936e20035ba8ff8b754fce1b063d90312d11d79fd3341fe" exitCode=0
Sep 12 02:40:47.214562 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:47.214386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7slnk" event={"ID":"79c2dd42-3472-444a-8710-de01329994af","Type":"ContainerDied","Data":"3e556d2f7e3e2b0f9936e20035ba8ff8b754fce1b063d90312d11d79fd3341fe"}
Sep 12 02:40:48.219355 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.219322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7slnk" event={"ID":"79c2dd42-3472-444a-8710-de01329994af","Type":"ContainerStarted","Data":"7d05c1f4985658d91e566e893c6b37125c1a4883ed5277c92213252793f8b9c7"}
Sep 12 02:40:48.219841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.219369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7slnk" event={"ID":"79c2dd42-3472-444a-8710-de01329994af","Type":"ContainerStarted","Data":"ce460d3e55b9908dcea30f4cd18449bd7a350f95fe959441d2513e3153945e3d"}
Sep 12 02:40:48.239259 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.239211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7slnk" podStartSLOduration=2.23624378 podStartE2EDuration="3.239197218s" totalImagesPullingTime="932.826609ms" totalInitContainerRuntime="70.126829ms" isStatefulPod=true podCreationTimestamp="2026-09-12 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:40:48.238582437 +0000 UTC m=+266.521496018" watchObservedRunningTime="2026-09-12 02:40:48.239197218 +0000 UTC m=+266.522110798"
Sep 12 02:40:48.411250 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.411219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:48.513524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.513458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:48.613300 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.613272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:48.713088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.713061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:48.814556 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:48.814470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:49.405238 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:49.405177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:49.506218 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:49.506189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:50.225796 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:50.225750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:50.604640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:50.604569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:51.111086 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:51.111029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:51.160070 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:51.160019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:51.632332 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:51.632303    2579 patch_prober.go:28] interesting pod/image-registry-6fdf8bffd8-6278h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 02:40:51.632750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:51.632356    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h" podUID="cdd3e22f-3a82-4f78-9e5a-83ca725b240d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:40:51.701972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:51.701929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:53.168341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:53.168310    2579 patch_prober.go:28] interesting pod/image-registry-6fdf8bffd8-6278h container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 02:40:53.168726 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:53.168365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h" podUID="cdd3e22f-3a82-4f78-9e5a-83ca725b240d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:40:59.404641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:59.404605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:59.505171 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:59.505122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:59.603933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:59.603894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:40:59.891680 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:40:59.891626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:00.071306 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:00.071277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:00.124028 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:00.123993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:00.277716 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:00.277645    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:00.472907 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:00.472873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:00.526217 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:00.526194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:00.879345 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:00.879308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:01.040616 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:01.040574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:01.141182 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:01.141085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:01.632411 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:01.632380    2579 patch_prober.go:28] interesting pod/image-registry-6fdf8bffd8-6278h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 12 02:41:01.632784 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:01.632434    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h" podUID="cdd3e22f-3a82-4f78-9e5a-83ca725b240d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:41:01.690857 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:01.690815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:01.739996 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:01.739968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:02.894247 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:41:02.894187    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pgkgx" podUID="e80ade3b-dcdf-4c97-96ce-4d14057a7b64"
Sep 12 02:41:03.168626 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.168544    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fdf8bffd8-6278h"
Sep 12 02:41:03.265533 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.265503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:41:03.283368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.283335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:03.331397 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.331353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:03.401092 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.401066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:03.452233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.452177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:03.723629 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.723554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:03.776322 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:03.776294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:04.715974 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:04.715943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:04.767426 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:04.767402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:06.514253 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:06.514206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:41:06.516623 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:06.516603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e80ade3b-dcdf-4c97-96ce-4d14057a7b64-metrics-tls\") pod \"dns-default-pgkgx\" (UID: \"e80ade3b-dcdf-4c97-96ce-4d14057a7b64\") " pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:41:06.569496 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:06.569466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4279k\""
Sep 12 02:41:06.577587 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:06.577568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:41:06.844130 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:06.844053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgkgx"]
Sep 12 02:41:06.846902 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:41:06.846869    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80ade3b_dcdf_4c97_96ce_4d14057a7b64.slice/crio-c7e2532697bccbd2c3ac9098a2bfe3b807da8da9de6304f5c24f7ee7ec073cfd WatchSource:0}: Error finding container c7e2532697bccbd2c3ac9098a2bfe3b807da8da9de6304f5c24f7ee7ec073cfd: Status 404 returned error can't find the container with id c7e2532697bccbd2c3ac9098a2bfe3b807da8da9de6304f5c24f7ee7ec073cfd
Sep 12 02:41:07.278038 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:07.278002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgkgx" event={"ID":"e80ade3b-dcdf-4c97-96ce-4d14057a7b64","Type":"ContainerStarted","Data":"c7e2532697bccbd2c3ac9098a2bfe3b807da8da9de6304f5c24f7ee7ec073cfd"}
Sep 12 02:41:08.405743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:08.405701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:08.506091 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:08.506052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:09.286662 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:09.286625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgkgx" event={"ID":"e80ade3b-dcdf-4c97-96ce-4d14057a7b64","Type":"ContainerStarted","Data":"737423bd6d9b703d286399775c90c9102df51e6d5173013a1d249d81c6586fc0"}
Sep 12 02:41:09.286818 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:09.286677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgkgx" event={"ID":"e80ade3b-dcdf-4c97-96ce-4d14057a7b64","Type":"ContainerStarted","Data":"472a9fc8428e5cf52df4c6bcb631914b4ae893e1dfdeff770c5c7c922f03fd77"}
Sep 12 02:41:09.286818 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:09.286772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:41:09.306667 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:09.306627    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pgkgx" podStartSLOduration=251.690323809 podStartE2EDuration="4m13.30661398s" totalImagesPullingTime="1.616290171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:41:09.305526478 +0000 UTC m=+287.588440060" watchObservedRunningTime="2026-09-12 02:41:09.30661398 +0000 UTC m=+287.589527555"
Sep 12 02:41:13.790146 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:13.790115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:13.842394 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:13.842369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:14.783895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:14.783853    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" podUID="9657b2f1-2213-4f6d-814f-e0292cee6f9d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 02:41:15.312041 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:15.312007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:15.364688 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:15.364655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:17.395883 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:17.395853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:17.499715 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:17.499685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:18.728763 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:18.728729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:18.782098 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:18.782065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:19.291800 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:19.291774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgkgx"
Sep 12 02:41:19.494706 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:19.494669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:22.126267 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:22.126240    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:41:22.128406 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:22.128384    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:41:22.129905 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:22.129886    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 02:41:23.105698 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:23.105662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:23.206735 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:23.206694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:23.870727 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:23.870694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:23.928681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:23.928651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:24.783747 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:24.783709    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" podUID="9657b2f1-2213-4f6d-814f-e0292cee6f9d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 02:41:24.902853 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:24.902818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:26.702009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:26.701968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:28.400806 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:28.400779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:33.108883 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:33.108846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:33.160976 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:33.160934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:33.707644 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:33.707608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:33.758555 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:33.758528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:34.783812 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:34.783778    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" podUID="9657b2f1-2213-4f6d-814f-e0292cee6f9d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 02:41:34.784176 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:34.783858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw"
Sep 12 02:41:34.784405 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:34.784377    2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0526c05ce9087b714b9de814177db11a08a54772c650a898b3457adf164997ab"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 12 02:41:34.784440 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:34.784428    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" podUID="9657b2f1-2213-4f6d-814f-e0292cee6f9d" containerName="service-proxy" containerID="cri-o://0526c05ce9087b714b9de814177db11a08a54772c650a898b3457adf164997ab" gracePeriod=30
Sep 12 02:41:35.366800 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:35.366771    2579 generic.go:356] "Generic (PLEG): container finished" podID="9657b2f1-2213-4f6d-814f-e0292cee6f9d" containerID="0526c05ce9087b714b9de814177db11a08a54772c650a898b3457adf164997ab" exitCode=2
Sep 12 02:41:35.366963 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:35.366827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" event={"ID":"9657b2f1-2213-4f6d-814f-e0292cee6f9d","Type":"ContainerDied","Data":"0526c05ce9087b714b9de814177db11a08a54772c650a898b3457adf164997ab"}
Sep 12 02:41:35.366963 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:35.366859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5668c584b8-jqbhw" event={"ID":"9657b2f1-2213-4f6d-814f-e0292cee6f9d","Type":"ContainerStarted","Data":"9a78db583486287a3cdf520cf61df3248c654b0a43cb9bfa7d6130ba8d1b5eaa"}
Sep 12 02:41:48.738800 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:48.738764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:41:48.836861 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:41:48.836834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:04.685953 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:04.685919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:04.999128 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:04.999043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:12.700893 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:12.700857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:12.749466 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:12.749430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:12.845594 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:12.845528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:12.945618 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:12.945585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:13.434613 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:13.434582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:13.484010 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:13.483981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:17.776887 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:17.776857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:17.835246 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:17.835220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:23.725943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:23.725908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:23.830388 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:23.830352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:23.929912 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:23.929880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:42:43.758790 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:42:43.758756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:19.939674 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:19.939639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:19.989926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:19.989894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:31.838000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:31.837965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:31.889043 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:31.889016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:37.486337 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:37.486303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:37.548742 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:37.548718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:48.075686 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:48.075604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:48.125605 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:48.125577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:48.176203 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:48.176181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:48.225790 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:48.225760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:48.733856 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:48.733820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:48.832973 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:48.832942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:43:49.012957 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:43:49.012870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:30.459080 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:30.459043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:30.555563 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:30.555536    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:52.013733 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:52.013697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:52.750210 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:52.750176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:55.413111 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:55.413076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:55.513506 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:55.513474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:56.635980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:56.635946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:56.729916 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:56.729880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:57.090213 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:57.090132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:57.188614 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:57.188587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:59.294316 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:59.294289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:44:59.515336 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:44:59.515309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:02.210767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:02.210736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:02.315315 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:02.315235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:05.818595 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:05.818567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:05.918429 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:05.918400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:07.720024 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.719993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:07.976916 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.976841    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"]
Sep 12 02:45:07.979815 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.979800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:07.982742 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.982722    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 12 02:45:07.982861 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.982759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 12 02:45:07.982861 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.982780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-29942\""
Sep 12 02:45:07.986807 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:07.986788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"]
Sep 12 02:45:08.094005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.093975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2179bef7-da95-4c47-90e3-9c33e1c54093-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f68xg\" (UID: \"2179bef7-da95-4c47-90e3-9c33e1c54093\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.094005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.094004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gfh\" (UniqueName: \"kubernetes.io/projected/2179bef7-da95-4c47-90e3-9c33e1c54093-kube-api-access-72gfh\") pod \"cert-manager-cainjector-559cdd8c87-f68xg\" (UID: \"2179bef7-da95-4c47-90e3-9c33e1c54093\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.194364 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.194329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2179bef7-da95-4c47-90e3-9c33e1c54093-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f68xg\" (UID: \"2179bef7-da95-4c47-90e3-9c33e1c54093\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.194364 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.194365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gfh\" (UniqueName: \"kubernetes.io/projected/2179bef7-da95-4c47-90e3-9c33e1c54093-kube-api-access-72gfh\") pod \"cert-manager-cainjector-559cdd8c87-f68xg\" (UID: \"2179bef7-da95-4c47-90e3-9c33e1c54093\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.202364 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.202336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2179bef7-da95-4c47-90e3-9c33e1c54093-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-f68xg\" (UID: \"2179bef7-da95-4c47-90e3-9c33e1c54093\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.202577 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.202555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gfh\" (UniqueName: \"kubernetes.io/projected/2179bef7-da95-4c47-90e3-9c33e1c54093-kube-api-access-72gfh\") pod \"cert-manager-cainjector-559cdd8c87-f68xg\" (UID: \"2179bef7-da95-4c47-90e3-9c33e1c54093\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.288374 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.288299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"
Sep 12 02:45:08.554547 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.554476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-f68xg"]
Sep 12 02:45:08.557289 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:45:08.557263    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2179bef7_da95_4c47_90e3_9c33e1c54093.slice/crio-2cc3d4b678ca331869ff2fc784fb86078209f68707b8e0ea1b2cfab12f7dd5cc WatchSource:0}: Error finding container 2cc3d4b678ca331869ff2fc784fb86078209f68707b8e0ea1b2cfab12f7dd5cc: Status 404 returned error can't find the container with id 2cc3d4b678ca331869ff2fc784fb86078209f68707b8e0ea1b2cfab12f7dd5cc
Sep 12 02:45:08.559322 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:08.559305    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:45:09.047308 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:09.047279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg" event={"ID":"2179bef7-da95-4c47-90e3-9c33e1c54093","Type":"ContainerStarted","Data":"2cc3d4b678ca331869ff2fc784fb86078209f68707b8e0ea1b2cfab12f7dd5cc"}
Sep 12 02:45:12.059980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:12.059887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg" event={"ID":"2179bef7-da95-4c47-90e3-9c33e1c54093","Type":"ContainerStarted","Data":"5bbf9ed3dafca06efb7a191a1345b197fdaae654f6eb2bd7466444f8d78c51e6"}
Sep 12 02:45:12.077262 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:12.077204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-f68xg" podStartSLOduration=1.957390544 podStartE2EDuration="5.077181701s" totalImagesPullingTime="3.119791157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:12.075548412 +0000 UTC m=+530.358461991" watchObservedRunningTime="2026-09-12 02:45:12.077181701 +0000 UTC m=+530.360095283"
Sep 12 02:45:12.109793 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:12.109767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:12.209615 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:12.209585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:12.438731 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:12.438683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:12.539214 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:12.539170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:13.776741 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:13.776691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:19.743723 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:19.743693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:28.048990 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:28.048958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:28.171815 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:28.171784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:28.278859 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:28.278826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:28.386765 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:28.386734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:28.834597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:28.834412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.087780 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.087690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.191712 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.191682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.207235 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.207208    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"]
Sep 12 02:45:29.211599 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.211575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.214548 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.214526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 02:45:29.214897 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.214883    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 02:45:29.215029 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.215012    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 02:45:29.215360 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.215348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 02:45:29.215556 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.215540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dx684\""
Sep 12 02:45:29.228521 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.228496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"]
Sep 12 02:45:29.339039 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.338947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.344039 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.344019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.344166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.344055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwqz\" (UniqueName: \"kubernetes.io/projected/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-kube-api-access-2gwqz\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.344166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.344077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-manifests\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.344166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.344099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-tmp\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.344166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.344114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-webhook-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.440617 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.440585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.445338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.445319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwqz\" (UniqueName: \"kubernetes.io/projected/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-kube-api-access-2gwqz\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.445473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.445354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-manifests\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.445473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.445387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-tmp\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.445473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.445413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-webhook-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.445636 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.445507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.445827 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.445801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-manifests\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.447911 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.447881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-tmp\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.448005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.447989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-webhook-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.448061 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.448006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.453504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.453482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwqz\" (UniqueName: \"kubernetes.io/projected/b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf-kube-api-access-2gwqz\") pod \"opendatahub-operator-controller-manager-774995dcdd-24wkz\" (UID: \"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf\") " pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.521063 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.521032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:29.548294 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.548264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.801980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.801945    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"]
Sep 12 02:45:29.805422 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:45:29.805380    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44d2542_e2ce_4d5c_baaa_e9f6e308ffaf.slice/crio-d0ad0eafcb17c2292136229d2fcc0f7bb9de549744bb78c2458dee36faac6b97 WatchSource:0}: Error finding container d0ad0eafcb17c2292136229d2fcc0f7bb9de549744bb78c2458dee36faac6b97: Status 404 returned error can't find the container with id d0ad0eafcb17c2292136229d2fcc0f7bb9de549744bb78c2458dee36faac6b97
Sep 12 02:45:29.860938 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.860907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:29.963862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:29.963829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:30.117019 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:30.116985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz" event={"ID":"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf","Type":"ContainerStarted","Data":"d0ad0eafcb17c2292136229d2fcc0f7bb9de549744bb78c2458dee36faac6b97"}
Sep 12 02:45:33.129244 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:33.129211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz" event={"ID":"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf","Type":"ContainerStarted","Data":"383b59fddd6c484917d1e51da143abd9276a28f865da0b9d103edcec6b5ba78f"}
Sep 12 02:45:34.134861 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:34.134829    2579 generic.go:356] "Generic (PLEG): container finished" podID="b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf" containerID="383b59fddd6c484917d1e51da143abd9276a28f865da0b9d103edcec6b5ba78f" exitCode=0
Sep 12 02:45:34.135239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:34.134918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz" event={"ID":"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf","Type":"ContainerDied","Data":"383b59fddd6c484917d1e51da143abd9276a28f865da0b9d103edcec6b5ba78f"}
Sep 12 02:45:35.140017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:35.139982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz" event={"ID":"b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf","Type":"ContainerStarted","Data":"7a413ca7e478f3456a1fc8e58d92c6824b3c21be4b79f63a2106a1ace9e7099a"}
Sep 12 02:45:35.140423 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:35.140108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:35.161110 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:35.161056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz" podStartSLOduration=1.90453699 podStartE2EDuration="6.161038097s" totalImagesPullingTime="3.253065371s" totalInitContainerRuntime="1.003435736s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:35.158879896 +0000 UTC m=+553.441793475" watchObservedRunningTime="2026-09-12 02:45:35.161038097 +0000 UTC m=+553.443951677"
Sep 12 02:45:41.047492 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:41.047438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:41.146625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:41.146572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:46.145956 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:46.145930    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-774995dcdd-24wkz"
Sep 12 02:45:47.980506 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:47.980457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:48.087163 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:48.087133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:48.946700 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:48.946671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:49.046925 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:49.046899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:50.943951 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.943922    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:50.980329 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.980293    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"]
Sep 12 02:45:50.983400 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.983378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:50.986138 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.986121    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 12 02:45:50.986138 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.986129    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 02:45:50.987464 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.987433    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 02:45:50.987560 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.987488    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 12 02:45:50.987560 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.987506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sll9c\""
Sep 12 02:45:50.993758 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:50.993738    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"]
Sep 12 02:45:51.101183 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.101146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-tls-certs\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.101347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.101214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-tmp\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.101347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.101237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zbm\" (UniqueName: \"kubernetes.io/projected/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-kube-api-access-b2zbm\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.202344 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.202262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-tmp\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.202344 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.202307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zbm\" (UniqueName: \"kubernetes.io/projected/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-kube-api-access-b2zbm\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.202548 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.202436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-tls-certs\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.204558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.204531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-tmp\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.204837 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.204818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-tls-certs\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.210859 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.210833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zbm\" (UniqueName: \"kubernetes.io/projected/7b38aa0d-1f5c-4f31-945b-fb9eec484e4d-kube-api-access-b2zbm\") pod \"kube-auth-proxy-84dbf54759-7mdcm\" (UID: \"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d\") " pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.226124 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.226078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:51.293397 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.293366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"
Sep 12 02:45:51.564931 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:51.564812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm"]
Sep 12 02:45:51.567388 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:45:51.567361    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b38aa0d_1f5c_4f31_945b_fb9eec484e4d.slice/crio-b01ed6ad30c4b8aeb6fa412539bf7a22c2599967d4ea2d176ad826c0aa0d0a7f WatchSource:0}: Error finding container b01ed6ad30c4b8aeb6fa412539bf7a22c2599967d4ea2d176ad826c0aa0d0a7f: Status 404 returned error can't find the container with id b01ed6ad30c4b8aeb6fa412539bf7a22c2599967d4ea2d176ad826c0aa0d0a7f
Sep 12 02:45:52.089247 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.089187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:52.092968 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.092933    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"]
Sep 12 02:45:52.095901 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.095880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.098686 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.098667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 12 02:45:52.098867 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.098761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4vqct\""
Sep 12 02:45:52.098867 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.098667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 12 02:45:52.099044 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.098720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 12 02:45:52.099044 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.098701    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 12 02:45:52.108103 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.108074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"]
Sep 12 02:45:52.152432 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.152393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:52.200720 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.200675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm" event={"ID":"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d","Type":"ContainerStarted","Data":"b01ed6ad30c4b8aeb6fa412539bf7a22c2599967d4ea2d176ad826c0aa0d0a7f"}
Sep 12 02:45:52.212248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.212248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszsk\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-kube-api-access-fszsk\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.212248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.212248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb139b-9ac8-4cff-87bd-021e91d65a95-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.212528 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.212528 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.212528 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.212324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.312732 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.312915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.312915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.312915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszsk\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-kube-api-access-fszsk\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.312915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.313136 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb139b-9ac8-4cff-87bd-021e91d65a95-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.313136 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.312945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.313733 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.313708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.316532 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.316507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb139b-9ac8-4cff-87bd-021e91d65a95-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.316766 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.316688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.317124 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.317104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.317552 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.317527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.321509 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.321488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.321916 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.321876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszsk\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-kube-api-access-fszsk\") pod \"istiod-openshift-gateway-7b7995d5cd-d7bxq\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.409070 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.409040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:52.698385 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.698304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"]
Sep 12 02:45:52.702354 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:45:52.702324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbb139b_9ac8_4cff_87bd_021e91d65a95.slice/crio-0fc1424f91231ff968c02318248fdbfc342abf0235921b72be7fb63be9913a78 WatchSource:0}: Error finding container 0fc1424f91231ff968c02318248fdbfc342abf0235921b72be7fb63be9913a78: Status 404 returned error can't find the container with id 0fc1424f91231ff968c02318248fdbfc342abf0235921b72be7fb63be9913a78
Sep 12 02:45:52.920791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:52.920754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:53.207887 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:53.207817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" event={"ID":"8dbb139b-9ac8-4cff-87bd-021e91d65a95","Type":"ContainerStarted","Data":"0fc1424f91231ff968c02318248fdbfc342abf0235921b72be7fb63be9913a78"}
Sep 12 02:45:55.218423 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:55.218381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm" event={"ID":"7b38aa0d-1f5c-4f31-945b-fb9eec484e4d","Type":"ContainerStarted","Data":"b2a81f4f272a2678c3f615f74cdd3104f593fdc036344fc056173514f082b9a1"}
Sep 12 02:45:55.235483 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:55.235424    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84dbf54759-7mdcm" podStartSLOduration=2.654756222 podStartE2EDuration="5.235407932s" totalImagesPullingTime="2.58065171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:55.234040257 +0000 UTC m=+573.516953838" watchObservedRunningTime="2026-09-12 02:45:55.235407932 +0000 UTC m=+573.518321511"
Sep 12 02:45:55.913558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:55.913519    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:45:55.913665 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:55.913597    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:45:56.224105 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:56.224008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" event={"ID":"8dbb139b-9ac8-4cff-87bd-021e91d65a95","Type":"ContainerStarted","Data":"cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486"}
Sep 12 02:45:56.224541 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:56.224183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:45:56.225888 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:56.225859    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-d7bxq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:45:56.225997 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:56.225905    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:45:56.246735 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:56.246697    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" podStartSLOduration=1.039120658 podStartE2EDuration="4.246683886s" totalImagesPullingTime="3.207563228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:45:56.244916659 +0000 UTC m=+574.527830239" watchObservedRunningTime="2026-09-12 02:45:56.246683886 +0000 UTC m=+574.529597465"
Sep 12 02:45:57.166736 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:57.166696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:57.228030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:57.227992    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-d7bxq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:45:57.228508 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:57.228046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:45:57.270014 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:57.269981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:57.760132 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:57.760095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:57.948187 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:57.948151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:58.042341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:58.042268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:59.056990 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.056959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:59.110358 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.110327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:59.211705 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.211674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:59.455207 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.455179    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vbm64"]
Sep 12 02:45:59.459463 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.459428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:45:59.462086 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.462063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 12 02:45:59.468154 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.468126    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vbm64"]
Sep 12 02:45:59.472933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.472910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:59.573214 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.573158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:45:59.576033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.576010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b1a0db-5cb0-4190-9d58-0377565062e8-tls-certs\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:45:59.576158 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.576047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlf6j\" (UniqueName: \"kubernetes.io/projected/47b1a0db-5cb0-4190-9d58-0377565062e8-kube-api-access-qlf6j\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:45:59.676963 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.676921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b1a0db-5cb0-4190-9d58-0377565062e8-tls-certs\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:45:59.676963 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.676963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlf6j\" (UniqueName: \"kubernetes.io/projected/47b1a0db-5cb0-4190-9d58-0377565062e8-kube-api-access-qlf6j\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:45:59.677199 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:45:59.677060    2579 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 12 02:45:59.677199 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:45:59.677128    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b1a0db-5cb0-4190-9d58-0377565062e8-tls-certs podName:47b1a0db-5cb0-4190-9d58-0377565062e8 nodeName:}" failed. No retries permitted until 2026-09-12 02:46:00.177111879 +0000 UTC m=+578.460025442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/47b1a0db-5cb0-4190-9d58-0377565062e8-tls-certs") pod "model-serving-api-54d89bd5fc-vbm64" (UID: "47b1a0db-5cb0-4190-9d58-0377565062e8") : secret "model-serving-api-tls" not found
Sep 12 02:45:59.685579 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.685556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlf6j\" (UniqueName: \"kubernetes.io/projected/47b1a0db-5cb0-4190-9d58-0377565062e8-kube-api-access-qlf6j\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:45:59.846843 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:45:59.846759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:00.181624 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:00.181579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b1a0db-5cb0-4190-9d58-0377565062e8-tls-certs\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:46:00.184167 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:00.184128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b1a0db-5cb0-4190-9d58-0377565062e8-tls-certs\") pod \"model-serving-api-54d89bd5fc-vbm64\" (UID: \"47b1a0db-5cb0-4190-9d58-0377565062e8\") " pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:46:00.228666 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:00.228639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:46:00.370857 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:00.370827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:46:00.640004 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:00.639976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vbm64"]
Sep 12 02:46:00.642611 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:46:00.642584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b1a0db_5cb0_4190_9d58_0377565062e8.slice/crio-ea59e5b44c13698cf4504103e030629291d4f969dd334a7d126d40f196a47ebc WatchSource:0}: Error finding container ea59e5b44c13698cf4504103e030629291d4f969dd334a7d126d40f196a47ebc: Status 404 returned error can't find the container with id ea59e5b44c13698cf4504103e030629291d4f969dd334a7d126d40f196a47ebc
Sep 12 02:46:01.242686 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:01.242656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64" event={"ID":"47b1a0db-5cb0-4190-9d58-0377565062e8","Type":"ContainerStarted","Data":"ea59e5b44c13698cf4504103e030629291d4f969dd334a7d126d40f196a47ebc"}
Sep 12 02:46:03.258093 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:03.258052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64" event={"ID":"47b1a0db-5cb0-4190-9d58-0377565062e8","Type":"ContainerStarted","Data":"eb4143900bb553541eaa4a94e60752b56460bbfe29b69e67e195d12e5a5be021"}
Sep 12 02:46:03.258519 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:03.258176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:46:03.275368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:03.275318    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64" podStartSLOduration=2.7338015479999997 podStartE2EDuration="4.275304556s" totalImagesPullingTime="1.541503008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:46:03.273784034 +0000 UTC m=+581.556697617" watchObservedRunningTime="2026-09-12 02:46:03.275304556 +0000 UTC m=+581.558218135"
Sep 12 02:46:08.101925 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.101895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:08.163861 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.163833    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"]
Sep 12 02:46:08.170782 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.170756    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"]
Sep 12 02:46:08.170921 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.170878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.172641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.172625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.173688 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.173671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f9lkp\""
Sep 12 02:46:08.173786 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.173709    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 12 02:46:08.175164 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.175124    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 12 02:46:08.175246 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.175229    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bq9bx\""
Sep 12 02:46:08.177916 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.177895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"]
Sep 12 02:46:08.180919 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.180902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"]
Sep 12 02:46:08.185500 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.185476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:08.232891 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.232855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7wqw\" (UniqueName: \"kubernetes.io/projected/89d4480e-ab4e-473a-8c1b-21da5a18f9cc-kube-api-access-c7wqw\") pod \"kserve-controller-manager-d6f4b65b8-m7rxq\" (UID: \"89d4480e-ab4e-473a-8c1b-21da5a18f9cc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.233065 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.233048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89d4480e-ab4e-473a-8c1b-21da5a18f9cc-cert\") pod \"kserve-controller-manager-d6f4b65b8-m7rxq\" (UID: \"89d4480e-ab4e-473a-8c1b-21da5a18f9cc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.276639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.276618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:08.334407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.334375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89d4480e-ab4e-473a-8c1b-21da5a18f9cc-cert\") pod \"kserve-controller-manager-d6f4b65b8-m7rxq\" (UID: \"89d4480e-ab4e-473a-8c1b-21da5a18f9cc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.334407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.334408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9707ce9-097b-4407-8bb0-ecdaf7261264-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sv2vd\" (UID: \"e9707ce9-097b-4407-8bb0-ecdaf7261264\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.334611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.334435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67sr\" (UniqueName: \"kubernetes.io/projected/e9707ce9-097b-4407-8bb0-ecdaf7261264-kube-api-access-l67sr\") pod \"llmisvc-controller-manager-777c6bd7d5-sv2vd\" (UID: \"e9707ce9-097b-4407-8bb0-ecdaf7261264\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.334611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.334483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wqw\" (UniqueName: \"kubernetes.io/projected/89d4480e-ab4e-473a-8c1b-21da5a18f9cc-kube-api-access-c7wqw\") pod \"kserve-controller-manager-d6f4b65b8-m7rxq\" (UID: \"89d4480e-ab4e-473a-8c1b-21da5a18f9cc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.337073 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.337043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89d4480e-ab4e-473a-8c1b-21da5a18f9cc-cert\") pod \"kserve-controller-manager-d6f4b65b8-m7rxq\" (UID: \"89d4480e-ab4e-473a-8c1b-21da5a18f9cc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.345293 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.345264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wqw\" (UniqueName: \"kubernetes.io/projected/89d4480e-ab4e-473a-8c1b-21da5a18f9cc-kube-api-access-c7wqw\") pod \"kserve-controller-manager-d6f4b65b8-m7rxq\" (UID: \"89d4480e-ab4e-473a-8c1b-21da5a18f9cc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.382200 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.382179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:08.435340 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.435313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9707ce9-097b-4407-8bb0-ecdaf7261264-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sv2vd\" (UID: \"e9707ce9-097b-4407-8bb0-ecdaf7261264\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.435473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.435350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67sr\" (UniqueName: \"kubernetes.io/projected/e9707ce9-097b-4407-8bb0-ecdaf7261264-kube-api-access-l67sr\") pod \"llmisvc-controller-manager-777c6bd7d5-sv2vd\" (UID: \"e9707ce9-097b-4407-8bb0-ecdaf7261264\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.437781 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.437754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9707ce9-097b-4407-8bb0-ecdaf7261264-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sv2vd\" (UID: \"e9707ce9-097b-4407-8bb0-ecdaf7261264\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.450638 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.450617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67sr\" (UniqueName: \"kubernetes.io/projected/e9707ce9-097b-4407-8bb0-ecdaf7261264-kube-api-access-l67sr\") pod \"llmisvc-controller-manager-777c6bd7d5-sv2vd\" (UID: \"e9707ce9-097b-4407-8bb0-ecdaf7261264\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.483427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.483408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:08.490376 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.490362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:08.752399 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.752329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"]
Sep 12 02:46:08.755359 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:46:08.755335    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d4480e_ab4e_473a_8c1b_21da5a18f9cc.slice/crio-a5e534d8410067999e233dcdd931bbf7e5ae5c13c4704068f1eea90c14025d34 WatchSource:0}: Error finding container a5e534d8410067999e233dcdd931bbf7e5ae5c13c4704068f1eea90c14025d34: Status 404 returned error can't find the container with id a5e534d8410067999e233dcdd931bbf7e5ae5c13c4704068f1eea90c14025d34
Sep 12 02:46:08.763729 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:08.763687    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"]
Sep 12 02:46:08.765973 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:46:08.765949    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9707ce9_097b_4407_8bb0_ecdaf7261264.slice/crio-bcfd76673aedb77cddd6e7ab8a27421c53f8eb42eeed6e54c4864eb5fd7f896a WatchSource:0}: Error finding container bcfd76673aedb77cddd6e7ab8a27421c53f8eb42eeed6e54c4864eb5fd7f896a: Status 404 returned error can't find the container with id bcfd76673aedb77cddd6e7ab8a27421c53f8eb42eeed6e54c4864eb5fd7f896a
Sep 12 02:46:09.279177 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.279145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd" event={"ID":"e9707ce9-097b-4407-8bb0-ecdaf7261264","Type":"ContainerStarted","Data":"bcfd76673aedb77cddd6e7ab8a27421c53f8eb42eeed6e54c4864eb5fd7f896a"}
Sep 12 02:46:09.280190 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.280168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq" event={"ID":"89d4480e-ab4e-473a-8c1b-21da5a18f9cc","Type":"ContainerStarted","Data":"a5e534d8410067999e233dcdd931bbf7e5ae5c13c4704068f1eea90c14025d34"}
Sep 12 02:46:09.361435 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.361400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:09.489742 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.489698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:09.559873 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.559787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:09.664876 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.664846    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"]
Sep 12 02:46:09.665133 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:09.665096    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerName="discovery" containerID="cri-o://cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486" gracePeriod=30
Sep 12 02:46:10.084645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.084623    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:46:10.148464 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.148421    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fszsk\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-kube-api-access-fszsk\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.148633 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.148518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-token\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.148633 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.148542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-kubeconfig\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.148633 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.148570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-ca-configmap\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.148975 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.148949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:46:10.150825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.150803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-token" (OuterVolumeSpecName: "istio-token") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:46:10.150825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.150815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:46:10.150970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.150853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-kube-api-access-fszsk" (OuterVolumeSpecName: "kube-api-access-fszsk") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "kube-api-access-fszsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:46:10.249305 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb139b-9ac8-4cff-87bd-021e91d65a95-local-certs\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.249488 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-dns-cert\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.249488 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249348    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-cacerts\") pod \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\" (UID: \"8dbb139b-9ac8-4cff-87bd-021e91d65a95\") "
Sep 12 02:46:10.249613 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249575    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:10.249613 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249591    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-kubeconfig\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:10.249613 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249604    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-ca-configmap\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:10.249737 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.249616    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszsk\" (UniqueName: \"kubernetes.io/projected/8dbb139b-9ac8-4cff-87bd-021e91d65a95-kube-api-access-fszsk\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:10.252125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.252070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb139b-9ac8-4cff-87bd-021e91d65a95-local-certs" (OuterVolumeSpecName: "local-certs") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:46:10.252358 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.252324    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-cacerts" (OuterVolumeSpecName: "cacerts") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:46:10.252514 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.252491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8dbb139b-9ac8-4cff-87bd-021e91d65a95" (UID: "8dbb139b-9ac8-4cff-87bd-021e91d65a95"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:46:10.286314 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.286246    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerID="cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486" exitCode=0
Sep 12 02:46:10.286314 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.286304    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"
Sep 12 02:46:10.286825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.286360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" event={"ID":"8dbb139b-9ac8-4cff-87bd-021e91d65a95","Type":"ContainerDied","Data":"cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486"}
Sep 12 02:46:10.286825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.286406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq" event={"ID":"8dbb139b-9ac8-4cff-87bd-021e91d65a95","Type":"ContainerDied","Data":"0fc1424f91231ff968c02318248fdbfc342abf0235921b72be7fb63be9913a78"}
Sep 12 02:46:10.286825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.286524    2579 scope.go:117] "RemoveContainer" containerID="cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486"
Sep 12 02:46:10.297699 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.297678    2579 scope.go:117] "RemoveContainer" containerID="cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486"
Sep 12 02:46:10.298107 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:46:10.298026    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486\": container with ID starting with cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486 not found: ID does not exist" containerID="cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486"
Sep 12 02:46:10.298107 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.298063    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486"} err="failed to get container status \"cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486\": rpc error: code = NotFound desc = could not find container \"cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486\": container with ID starting with cc04c6727394389a29b6ed30282d992c84b0de47da943e9ca9af51bec40b8486 not found: ID does not exist"
Sep 12 02:46:10.311848 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.311816    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"]
Sep 12 02:46:10.316301 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.316277    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d7bxq"]
Sep 12 02:46:10.349918 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.349900    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb139b-9ac8-4cff-87bd-021e91d65a95-local-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:10.349918 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.349919    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-istio-csr-dns-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:10.350031 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:10.349928    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8dbb139b-9ac8-4cff-87bd-021e91d65a95-cacerts\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:46:11.295005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:11.294969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:11.396279 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:11.396252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:11.453999 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:11.453973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:12.010933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.010904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:12.070702 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.070670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:12.244256 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.244223    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" path="/var/lib/kubelet/pods/8dbb139b-9ac8-4cff-87bd-021e91d65a95/volumes"
Sep 12 02:46:12.298132 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.297890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd" event={"ID":"e9707ce9-097b-4407-8bb0-ecdaf7261264","Type":"ContainerStarted","Data":"be65bc0d7add062c20a7ff8439d06cfd260c8b45a51ccfba0013063a46603245"}
Sep 12 02:46:12.298132 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.298010    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:12.299574 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.299524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq" event={"ID":"89d4480e-ab4e-473a-8c1b-21da5a18f9cc","Type":"ContainerStarted","Data":"8ce89b6d913e4196cadbdc6c5d52c44fac6ac970b3593875940aaa90bf68be0b"}
Sep 12 02:46:12.299715 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.299634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:46:12.314805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.314754    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd" podStartSLOduration=1.598358026 podStartE2EDuration="4.314737854s" totalImagesPullingTime="2.716379828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:46:12.312912816 +0000 UTC m=+590.595826397" watchObservedRunningTime="2026-09-12 02:46:12.314737854 +0000 UTC m=+590.597651433"
Sep 12 02:46:12.329044 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:12.329007    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq" podStartSLOduration=1.60490558 podStartE2EDuration="4.328994405s" totalImagesPullingTime="2.724088825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:46:12.327675204 +0000 UTC m=+590.610588785" watchObservedRunningTime="2026-09-12 02:46:12.328994405 +0000 UTC m=+590.611907984"
Sep 12 02:46:14.265355 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:14.265324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vbm64"
Sep 12 02:46:15.723489 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:15.723409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:15.821779 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:15.821741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:21.825648 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:21.825617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:21.876050 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:21.876024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:21.925380 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:21.925349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:46:22.151020 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:22.150999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:46:22.151357 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:22.151338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:46:43.305639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:43.305560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sv2vd"
Sep 12 02:46:43.313862 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:46:43.308539    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m7rxq"
Sep 12 02:47:00.332118 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:00.332079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:00.420273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:00.420235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:12.959743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:12.959680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:13.752161 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:13.752131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:16.221943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.221915    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dhdgg"]
Sep 12 02:47:16.222390 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.222182    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerName="discovery"
Sep 12 02:47:16.222390 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.222192    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerName="discovery"
Sep 12 02:47:16.222390 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.222246    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbb139b-9ac8-4cff-87bd-021e91d65a95" containerName="discovery"
Sep 12 02:47:16.224888 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.224870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:16.228365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.228341    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 02:47:16.228731 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.228711    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 02:47:16.229648 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.229625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zfw5z\""
Sep 12 02:47:16.237327 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.237309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dhdgg"]
Sep 12 02:47:16.318527 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.318489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jrx\" (UniqueName: \"kubernetes.io/projected/75a3c81c-a0b3-44ef-83a3-3f36abade8c6-kube-api-access-b7jrx\") pod \"authorino-operator-76b478bd78-dhdgg\" (UID: \"75a3c81c-a0b3-44ef-83a3-3f36abade8c6\") " pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:16.419234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.419203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jrx\" (UniqueName: \"kubernetes.io/projected/75a3c81c-a0b3-44ef-83a3-3f36abade8c6-kube-api-access-b7jrx\") pod \"authorino-operator-76b478bd78-dhdgg\" (UID: \"75a3c81c-a0b3-44ef-83a3-3f36abade8c6\") " pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:16.430353 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.430331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jrx\" (UniqueName: \"kubernetes.io/projected/75a3c81c-a0b3-44ef-83a3-3f36abade8c6-kube-api-access-b7jrx\") pod \"authorino-operator-76b478bd78-dhdgg\" (UID: \"75a3c81c-a0b3-44ef-83a3-3f36abade8c6\") " pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:16.533726 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.533652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:16.804639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.804617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dhdgg"]
Sep 12 02:47:16.807287 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:47:16.807252    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a3c81c_a0b3_44ef_83a3_3f36abade8c6.slice/crio-ad8191548509968ed8af41d82d18239654afc65dc09637935acdf9199c67c092 WatchSource:0}: Error finding container ad8191548509968ed8af41d82d18239654afc65dc09637935acdf9199c67c092: Status 404 returned error can't find the container with id ad8191548509968ed8af41d82d18239654afc65dc09637935acdf9199c67c092
Sep 12 02:47:16.958286 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:16.958247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:17.207365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:17.207333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:17.304645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:17.304620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:17.540793 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:17.540708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg" event={"ID":"75a3c81c-a0b3-44ef-83a3-3f36abade8c6","Type":"ContainerStarted","Data":"ad8191548509968ed8af41d82d18239654afc65dc09637935acdf9199c67c092"}
Sep 12 02:47:18.425261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.425228    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"]
Sep 12 02:47:18.428814 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.428765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:18.431520 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.431500    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xl28q\""
Sep 12 02:47:18.441683 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.441664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"]
Sep 12 02:47:18.536203 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.536176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbg6z\" (UniqueName: \"kubernetes.io/projected/f9469af4-8ef8-4583-8eb9-20f7eddaf2af-kube-api-access-dbg6z\") pod \"limitador-operator-controller-manager-74495c69f8-zb9mg\" (UID: \"f9469af4-8ef8-4583-8eb9-20f7eddaf2af\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:18.637064 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.637027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbg6z\" (UniqueName: \"kubernetes.io/projected/f9469af4-8ef8-4583-8eb9-20f7eddaf2af-kube-api-access-dbg6z\") pod \"limitador-operator-controller-manager-74495c69f8-zb9mg\" (UID: \"f9469af4-8ef8-4583-8eb9-20f7eddaf2af\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:18.645765 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.645732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbg6z\" (UniqueName: \"kubernetes.io/projected/f9469af4-8ef8-4583-8eb9-20f7eddaf2af-kube-api-access-dbg6z\") pod \"limitador-operator-controller-manager-74495c69f8-zb9mg\" (UID: \"f9469af4-8ef8-4583-8eb9-20f7eddaf2af\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:18.740821 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.740744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:18.948591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:18.948556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:19.202646 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.202607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:19.357758 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.357723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:19.551898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.551873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg" event={"ID":"75a3c81c-a0b3-44ef-83a3-3f36abade8c6","Type":"ContainerStarted","Data":"3a903fb643be563f2b2e38f57b684c04ca12670665581b0f2abdf66028e78cdc"}
Sep 12 02:47:19.552154 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.551988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:19.569864 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.569818    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg" podStartSLOduration=0.892075337 podStartE2EDuration="3.569803289s" totalImagesPullingTime="2.677727952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:19.569120487 +0000 UTC m=+657.852034065" watchObservedRunningTime="2026-09-12 02:47:19.569803289 +0000 UTC m=+657.852716868"
Sep 12 02:47:19.687037 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.686827    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"]
Sep 12 02:47:19.696465 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:47:19.696422    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9469af4_8ef8_4583_8eb9_20f7eddaf2af.slice/crio-98b4a7c02e2fc42450294bcb20a319173428335548ba09cd6531926e2bc9b71d WatchSource:0}: Error finding container 98b4a7c02e2fc42450294bcb20a319173428335548ba09cd6531926e2bc9b71d: Status 404 returned error can't find the container with id 98b4a7c02e2fc42450294bcb20a319173428335548ba09cd6531926e2bc9b71d
Sep 12 02:47:19.958504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:19.958466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:20.206082 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:20.206050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:20.309698 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:20.309618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:20.558641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:20.558576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg" event={"ID":"f9469af4-8ef8-4583-8eb9-20f7eddaf2af","Type":"ContainerStarted","Data":"98b4a7c02e2fc42450294bcb20a319173428335548ba09cd6531926e2bc9b71d"}
Sep 12 02:47:21.563801 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:21.563757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg" event={"ID":"f9469af4-8ef8-4583-8eb9-20f7eddaf2af","Type":"ContainerStarted","Data":"7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465"}
Sep 12 02:47:21.564173 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:21.563837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:21.585381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:21.585340    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg" podStartSLOduration=1.837145546 podStartE2EDuration="3.585324639s" totalImagesPullingTime="1.748179093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:21.583939466 +0000 UTC m=+659.866853045" watchObservedRunningTime="2026-09-12 02:47:21.585324639 +0000 UTC m=+659.868238219"
Sep 12 02:47:22.651405 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.651378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"]
Sep 12 02:47:22.654476 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.654442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.657122 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.657104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjdc8\""
Sep 12 02:47:22.665772 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.665756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"]
Sep 12 02:47:22.770466 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.770407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5tc5\" (UniqueName: \"kubernetes.io/projected/30c50386-8cf2-4724-9374-1075a2ac4864-kube-api-access-x5tc5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.770466 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.770437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c50386-8cf2-4724-9374-1075a2ac4864-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.871166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.871124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5tc5\" (UniqueName: \"kubernetes.io/projected/30c50386-8cf2-4724-9374-1075a2ac4864-kube-api-access-x5tc5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.871166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.871160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c50386-8cf2-4724-9374-1075a2ac4864-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.871587 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.871570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c50386-8cf2-4724-9374-1075a2ac4864-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.884835 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.884808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5tc5\" (UniqueName: \"kubernetes.io/projected/30c50386-8cf2-4724-9374-1075a2ac4864-kube-api-access-x5tc5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:22.963832 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:22.963757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:23.171983 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:23.171947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:23.238376 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:23.238352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"]
Sep 12 02:47:23.241144 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:47:23.241117    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c50386_8cf2_4724_9374_1075a2ac4864.slice/crio-d2af180a1f72cc1111274717ca6167b08062c7ae7f64830bcae10c25f6993b2a WatchSource:0}: Error finding container d2af180a1f72cc1111274717ca6167b08062c7ae7f64830bcae10c25f6993b2a: Status 404 returned error can't find the container with id d2af180a1f72cc1111274717ca6167b08062c7ae7f64830bcae10c25f6993b2a
Sep 12 02:47:23.571758 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:23.571682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl" event={"ID":"30c50386-8cf2-4724-9374-1075a2ac4864","Type":"ContainerStarted","Data":"d2af180a1f72cc1111274717ca6167b08062c7ae7f64830bcae10c25f6993b2a"}
Sep 12 02:47:24.706889 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:24.706854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:26.157667 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:26.157634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:26.254593 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:26.254551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:26.560464 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:26.560361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:26.623635 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:26.623608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:27.593127 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:27.593094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl" event={"ID":"30c50386-8cf2-4724-9374-1075a2ac4864","Type":"ContainerStarted","Data":"5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4"}
Sep 12 02:47:27.593513 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:27.593245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:27.619810 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:27.619762    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl" podStartSLOduration=1.8453304780000002 podStartE2EDuration="5.619748903s" totalImagesPullingTime="3.774418425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:27.61707224 +0000 UTC m=+665.899985817" watchObservedRunningTime="2026-09-12 02:47:27.619748903 +0000 UTC m=+665.902662483"
Sep 12 02:47:29.585094 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.585060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:29.588499 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.588473    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"]
Sep 12 02:47:29.591800 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.591778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.594589 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.594572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 12 02:47:29.594678 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.594594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sq5df\""
Sep 12 02:47:29.594678 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.594639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 12 02:47:29.600219 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.600200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"]
Sep 12 02:47:29.729286 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.729259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszxj\" (UniqueName: \"kubernetes.io/projected/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-kube-api-access-qszxj\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.729286 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.729286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.729480 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.729304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.773419 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.773397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:29.830604 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.830576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszxj\" (UniqueName: \"kubernetes.io/projected/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-kube-api-access-qszxj\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.830758 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.830613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.830758 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.830641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.831304 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.831282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.833135 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.833110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.838804 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.838762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszxj\" (UniqueName: \"kubernetes.io/projected/4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2-kube-api-access-qszxj\") pod \"kuadrant-console-plugin-fbdd4f445-5gmn5\" (UID: \"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:29.901078 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:29.901049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"
Sep 12 02:47:30.159147 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:30.159120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:30.166035 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:30.166009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5"]
Sep 12 02:47:30.168161 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:47:30.168136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4afb5ac9_4a91_40b3_8ab6_dcc1b04d36e2.slice/crio-dd2e09a093838ed79a3db3b0d446a092b5bcb2642c815a097362fbe28193224b WatchSource:0}: Error finding container dd2e09a093838ed79a3db3b0d446a092b5bcb2642c815a097362fbe28193224b: Status 404 returned error can't find the container with id dd2e09a093838ed79a3db3b0d446a092b5bcb2642c815a097362fbe28193224b
Sep 12 02:47:30.260443 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:30.260420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:30.560754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:30.560687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-dhdgg"
Sep 12 02:47:30.606970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:30.606929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5" event={"ID":"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2","Type":"ContainerStarted","Data":"dd2e09a093838ed79a3db3b0d446a092b5bcb2642c815a097362fbe28193224b"}
Sep 12 02:47:32.569583 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:32.569556    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:33.966401 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:33.966359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:34.066495 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:34.066459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:34.167294 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:34.167263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:35.632123 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:35.632088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5" event={"ID":"4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2","Type":"ContainerStarted","Data":"1a8434f79e12e204895a72b2c3ddf605e6241147f13b68efa12d378683ddbefe"}
Sep 12 02:47:35.649318 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:35.649277    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5gmn5" podStartSLOduration=2.070658996 podStartE2EDuration="6.649263595s" totalImagesPullingTime="4.578604599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:35.646928939 +0000 UTC m=+673.929842518" watchObservedRunningTime="2026-09-12 02:47:35.649263595 +0000 UTC m=+673.932177174"
Sep 12 02:47:38.599177 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:38.599144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:39.079685 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:39.079650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:39.174077 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:39.174046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:40.522931 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.522895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:40.622359 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.622330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:40.801426 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.801323    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"]
Sep 12 02:47:40.805754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.805733    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"]
Sep 12 02:47:40.805882 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.805868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:40.805923 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.805880    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" containerName="manager" containerID="cri-o://5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4" gracePeriod=2
Sep 12 02:47:40.814177 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.813611    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"]
Sep 12 02:47:40.816744 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.816706    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"]
Sep 12 02:47:40.816975 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:40.816921    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"30c50386-8cf2-4724-9374-1075a2ac4864\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\": pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:40.828262 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.828241    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"]
Sep 12 02:47:40.828487 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.828460    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" containerName="manager" containerID="cri-o://7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465" gracePeriod=2
Sep 12 02:47:40.840490 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.840439    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"]
Sep 12 02:47:40.846971 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:40.846940    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:40.849150 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:40.849123    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:40.851639 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:40.851615    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:40.857597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.857570    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"]
Sep 12 02:47:40.857880 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.857863    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" containerName="manager"
Sep 12 02:47:40.857880 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.857880    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" containerName="manager"
Sep 12 02:47:40.858057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.857911    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" containerName="manager"
Sep 12 02:47:40.858057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.857919    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" containerName="manager"
Sep 12 02:47:40.858057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.857992    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" containerName="manager"
Sep 12 02:47:40.858057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.858004    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" containerName="manager"
Sep 12 02:47:40.861009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.860992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:40.863475 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:40.863432    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:40.872239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.872218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"]
Sep 12 02:47:40.897840 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:40.897808    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:40.920293 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.920262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd1f1ae9-de06-471c-9c69-c460bccd57a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8mb5f\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:40.920406 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:40.920309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg44\" (UniqueName: \"kubernetes.io/projected/dd1f1ae9-de06-471c-9c69-c460bccd57a8-kube-api-access-cpg44\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8mb5f\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.021048 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.021007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnndm\" (UniqueName: \"kubernetes.io/projected/d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e-kube-api-access-jnndm\") pod \"limitador-operator-controller-manager-74495c69f8-rd7zb\" (UID: \"d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:41.021204 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.021127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd1f1ae9-de06-471c-9c69-c460bccd57a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8mb5f\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.021204 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.021159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg44\" (UniqueName: \"kubernetes.io/projected/dd1f1ae9-de06-471c-9c69-c460bccd57a8-kube-api-access-cpg44\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8mb5f\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.021615 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.021588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd1f1ae9-de06-471c-9c69-c460bccd57a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8mb5f\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.026755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.026732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:41.034107 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.034087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg44\" (UniqueName: \"kubernetes.io/projected/dd1f1ae9-de06-471c-9c69-c460bccd57a8-kube-api-access-cpg44\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-8mb5f\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.059662 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.059640    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:41.062703 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.062674    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.065167 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.065143    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:41.069222 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.069206    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.071674 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.071653    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:41.073607 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.073584    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.119980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.119958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.121593 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.121574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnndm\" (UniqueName: \"kubernetes.io/projected/d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e-kube-api-access-jnndm\") pod \"limitador-operator-controller-manager-74495c69f8-rd7zb\" (UID: \"d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:41.123068 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.123038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:41.129911 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.129894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnndm\" (UniqueName: \"kubernetes.io/projected/d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e-kube-api-access-jnndm\") pod \"limitador-operator-controller-manager-74495c69f8-rd7zb\" (UID: \"d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:41.221913 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.221885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5tc5\" (UniqueName: \"kubernetes.io/projected/30c50386-8cf2-4724-9374-1075a2ac4864-kube-api-access-x5tc5\") pod \"30c50386-8cf2-4724-9374-1075a2ac4864\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") "
Sep 12 02:47:41.222047 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.221931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c50386-8cf2-4724-9374-1075a2ac4864-extensions-socket-volume\") pod \"30c50386-8cf2-4724-9374-1075a2ac4864\" (UID: \"30c50386-8cf2-4724-9374-1075a2ac4864\") "
Sep 12 02:47:41.222047 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.221964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbg6z\" (UniqueName: \"kubernetes.io/projected/f9469af4-8ef8-4583-8eb9-20f7eddaf2af-kube-api-access-dbg6z\") pod \"f9469af4-8ef8-4583-8eb9-20f7eddaf2af\" (UID: \"f9469af4-8ef8-4583-8eb9-20f7eddaf2af\") "
Sep 12 02:47:41.222500 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.222438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c50386-8cf2-4724-9374-1075a2ac4864-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "30c50386-8cf2-4724-9374-1075a2ac4864" (UID: "30c50386-8cf2-4724-9374-1075a2ac4864"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:47:41.224185 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.224161    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9469af4-8ef8-4583-8eb9-20f7eddaf2af-kube-api-access-dbg6z" (OuterVolumeSpecName: "kube-api-access-dbg6z") pod "f9469af4-8ef8-4583-8eb9-20f7eddaf2af" (UID: "f9469af4-8ef8-4583-8eb9-20f7eddaf2af"). InnerVolumeSpecName "kube-api-access-dbg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:47:41.224268 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.224201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c50386-8cf2-4724-9374-1075a2ac4864-kube-api-access-x5tc5" (OuterVolumeSpecName: "kube-api-access-x5tc5") pod "30c50386-8cf2-4724-9374-1075a2ac4864" (UID: "30c50386-8cf2-4724-9374-1075a2ac4864"). InnerVolumeSpecName "kube-api-access-x5tc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:47:41.227330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.227313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:41.322958 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.322896    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbg6z\" (UniqueName: \"kubernetes.io/projected/f9469af4-8ef8-4583-8eb9-20f7eddaf2af-kube-api-access-dbg6z\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:47:41.322958 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.322921    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5tc5\" (UniqueName: \"kubernetes.io/projected/30c50386-8cf2-4724-9374-1075a2ac4864-kube-api-access-x5tc5\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:47:41.322958 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.322930    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c50386-8cf2-4724-9374-1075a2ac4864-extensions-socket-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:47:41.393636 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.393587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"]
Sep 12 02:47:41.397543 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:47:41.397512    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1f1ae9_de06_471c_9c69_c460bccd57a8.slice/crio-da0ee9bc65b597dcb44487b081ff4c45a9ad8f707b87a302fbb069b21e778f67 WatchSource:0}: Error finding container da0ee9bc65b597dcb44487b081ff4c45a9ad8f707b87a302fbb069b21e778f67: Status 404 returned error can't find the container with id da0ee9bc65b597dcb44487b081ff4c45a9ad8f707b87a302fbb069b21e778f67
Sep 12 02:47:41.494372 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.494345    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"]
Sep 12 02:47:41.497213 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:47:41.497185    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f5d4c4_c13b_41c3_8a7c_164094fc4f8e.slice/crio-b1020f4c65ce510d1d24887e56ccb6f8242d95e98c1010aea832602dbc3c30c2 WatchSource:0}: Error finding container b1020f4c65ce510d1d24887e56ccb6f8242d95e98c1010aea832602dbc3c30c2: Status 404 returned error can't find the container with id b1020f4c65ce510d1d24887e56ccb6f8242d95e98c1010aea832602dbc3c30c2
Sep 12 02:47:41.658522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.658490    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" containerID="7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465" exitCode=0
Sep 12 02:47:41.658929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.658574    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.658929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.658637    2579 scope.go:117] "RemoveContainer" containerID="7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465"
Sep 12 02:47:41.659981 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.659962    2579 generic.go:356] "Generic (PLEG): container finished" podID="30c50386-8cf2-4724-9374-1075a2ac4864" containerID="5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4" exitCode=0
Sep 12 02:47:41.660100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.660006    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:41.664677 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.664627    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.666403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.666308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f" event={"ID":"dd1f1ae9-de06-471c-9c69-c460bccd57a8","Type":"ContainerStarted","Data":"6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875"}
Sep 12 02:47:41.666403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.666365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f" event={"ID":"dd1f1ae9-de06-471c-9c69-c460bccd57a8","Type":"ContainerStarted","Data":"da0ee9bc65b597dcb44487b081ff4c45a9ad8f707b87a302fbb069b21e778f67"}
Sep 12 02:47:41.672841 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.667368    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:41.672841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.667792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:41.672841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.668712    2579 scope.go:117] "RemoveContainer" containerID="7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465"
Sep 12 02:47:41.672841 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.670102    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:41.673380 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.673339    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465\": container with ID starting with 7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465 not found: ID does not exist" containerID="7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465"
Sep 12 02:47:41.673484 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.673379    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465"} err="failed to get container status \"7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465\": rpc error: code = NotFound desc = could not find container \"7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465\": container with ID starting with 7a20c4ea244ddcef1b6ab027a13e46fc53bff54fb74c59101b5a19a4ae18e465 not found: ID does not exist"
Sep 12 02:47:41.673484 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.673403    2579 scope.go:117] "RemoveContainer" containerID="5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4"
Sep 12 02:47:41.675129 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.675086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb" event={"ID":"d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e","Type":"ContainerStarted","Data":"2222b49222fa4506172e037703171cee314a2e59baf43cfb0b752cace15faf41"}
Sep 12 02:47:41.675311 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.675285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb" event={"ID":"d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e","Type":"ContainerStarted","Data":"b1020f4c65ce510d1d24887e56ccb6f8242d95e98c1010aea832602dbc3c30c2"}
Sep 12 02:47:41.675511 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.675488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:41.682469 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.682434    2579 scope.go:117] "RemoveContainer" containerID="5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4"
Sep 12 02:47:41.682762 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.682745    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4\": container with ID starting with 5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4 not found: ID does not exist" containerID="5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4"
Sep 12 02:47:41.682816 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.682770    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4"} err="failed to get container status \"5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4\": rpc error: code = NotFound desc = could not find container \"5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4\": container with ID starting with 5b6b1c4e9da148522d4eff323b294b75f7bffcc4dfa4de8353d03116198344a4 not found: ID does not exist"
Sep 12 02:47:41.692192 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.692154    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f" podStartSLOduration=1.6921415450000001 podStartE2EDuration="1.692141545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:41.690870194 +0000 UTC m=+679.973783775" watchObservedRunningTime="2026-09-12 02:47:41.692141545 +0000 UTC m=+679.975055126"
Sep 12 02:47:41.692928 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.692905    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.711667 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:41.711628    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb" podStartSLOduration=1.7116161779999999 podStartE2EDuration="1.711616178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:47:41.710220576 +0000 UTC m=+679.993134162" watchObservedRunningTime="2026-09-12 02:47:41.711616178 +0000 UTC m=+679.994529760"
Sep 12 02:47:41.712244 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.712224    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:41.714099 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:41.714081    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:42.243673 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:42.243635    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zxbtl\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zxbtl"
Sep 12 02:47:42.245578 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:42.245559    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c50386-8cf2-4724-9374-1075a2ac4864" path="/var/lib/kubelet/pods/30c50386-8cf2-4724-9374-1075a2ac4864/volumes"
Sep 12 02:47:42.245764 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:47:42.245742    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-zb9mg\" is forbidden: User \"system:node:ip-10-0-128-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-128.ec2.internal' and this object" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-zb9mg"
Sep 12 02:47:42.245914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:42.245901    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9469af4-8ef8-4583-8eb9-20f7eddaf2af" path="/var/lib/kubelet/pods/f9469af4-8ef8-4583-8eb9-20f7eddaf2af/volumes"
Sep 12 02:47:46.012312 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:46.012286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:46.065725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:46.065696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:46.602486 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:46.602439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:46.702936 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:46.702907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:50.001364 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:50.001334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:50.101929 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:50.101898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:52.682727 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:52.682694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rd7zb"
Sep 12 02:47:52.683166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:52.682770    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:47:54.213854 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:54.213815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:54.313795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:54.313766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:54.413684 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:54.413651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:54.514963 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:54.514880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:54.615336 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:54.615304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:47:54.756187 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:47:54.756161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:00.454913 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:00.454840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:00.555263 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:00.555234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:05.238680 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.238647    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"]
Sep 12 02:48:05.239234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.239198    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f" podUID="dd1f1ae9-de06-471c-9c69-c460bccd57a8" containerName="manager" containerID="cri-o://6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875" gracePeriod=10
Sep 12 02:48:05.500601 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.500533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:48:05.605803 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.605764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd1f1ae9-de06-471c-9c69-c460bccd57a8-extensions-socket-volume\") pod \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") "
Sep 12 02:48:05.605803 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.605813    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpg44\" (UniqueName: \"kubernetes.io/projected/dd1f1ae9-de06-471c-9c69-c460bccd57a8-kube-api-access-cpg44\") pod \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\" (UID: \"dd1f1ae9-de06-471c-9c69-c460bccd57a8\") "
Sep 12 02:48:05.606187 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.606163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1f1ae9-de06-471c-9c69-c460bccd57a8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dd1f1ae9-de06-471c-9c69-c460bccd57a8" (UID: "dd1f1ae9-de06-471c-9c69-c460bccd57a8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:48:05.608082 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.608055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1f1ae9-de06-471c-9c69-c460bccd57a8-kube-api-access-cpg44" (OuterVolumeSpecName: "kube-api-access-cpg44") pod "dd1f1ae9-de06-471c-9c69-c460bccd57a8" (UID: "dd1f1ae9-de06-471c-9c69-c460bccd57a8"). InnerVolumeSpecName "kube-api-access-cpg44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:48:05.707113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.707076    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd1f1ae9-de06-471c-9c69-c460bccd57a8-extensions-socket-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:05.707113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.707106    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg44\" (UniqueName: \"kubernetes.io/projected/dd1f1ae9-de06-471c-9c69-c460bccd57a8-kube-api-access-cpg44\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:05.783655 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.783587    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd1f1ae9-de06-471c-9c69-c460bccd57a8" containerID="6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875" exitCode=0
Sep 12 02:48:05.783655 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.783634    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"
Sep 12 02:48:05.783840 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.783633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f" event={"ID":"dd1f1ae9-de06-471c-9c69-c460bccd57a8","Type":"ContainerDied","Data":"6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875"}
Sep 12 02:48:05.783840 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.783820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f" event={"ID":"dd1f1ae9-de06-471c-9c69-c460bccd57a8","Type":"ContainerDied","Data":"da0ee9bc65b597dcb44487b081ff4c45a9ad8f707b87a302fbb069b21e778f67"}
Sep 12 02:48:05.783942 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.783863    2579 scope.go:117] "RemoveContainer" containerID="6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875"
Sep 12 02:48:05.792278 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.792258    2579 scope.go:117] "RemoveContainer" containerID="6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875"
Sep 12 02:48:05.792556 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:48:05.792537    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875\": container with ID starting with 6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875 not found: ID does not exist" containerID="6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875"
Sep 12 02:48:05.792625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.792567    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875"} err="failed to get container status \"6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875\": rpc error: code = NotFound desc = could not find container \"6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875\": container with ID starting with 6923964f7f9ddc8aad31234cc4858701e351271a89af6b1605df8639d8c21875 not found: ID does not exist"
Sep 12 02:48:05.804279 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.804251    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"]
Sep 12 02:48:05.808083 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:05.808058    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-8mb5f"]
Sep 12 02:48:06.244895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:06.244866    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd1f1ae9-de06-471c-9c69-c460bccd57a8" path="/var/lib/kubelet/pods/dd1f1ae9-de06-471c-9c69-c460bccd57a8/volumes"
Sep 12 02:48:09.149893 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.149858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"]
Sep 12 02:48:09.150274 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.150147    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1f1ae9-de06-471c-9c69-c460bccd57a8" containerName="manager"
Sep 12 02:48:09.150274 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.150158    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1f1ae9-de06-471c-9c69-c460bccd57a8" containerName="manager"
Sep 12 02:48:09.150274 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.150208    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1f1ae9-de06-471c-9c69-c460bccd57a8" containerName="manager"
Sep 12 02:48:09.159006 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.158980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.161838 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.161817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hrd7f\""
Sep 12 02:48:09.161960 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.161885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 12 02:48:09.165974 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.165859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"]
Sep 12 02:48:09.177717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.177691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:09.234634 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234779 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j77t\" (UniqueName: \"kubernetes.io/projected/fd97812e-9629-4da1-a7e5-dccfcb749b45-kube-api-access-9j77t\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234779 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234779 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234779 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd97812e-9629-4da1-a7e5-dccfcb749b45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.234917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.235007 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.234920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.332550 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.332512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:09.335631 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd97812e-9629-4da1-a7e5-dccfcb749b45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335694 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335694 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335796 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335796 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335796 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j77t\" (UniqueName: \"kubernetes.io/projected/fd97812e-9629-4da1-a7e5-dccfcb749b45-kube-api-access-9j77t\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.335943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.335896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.336265 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.336240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.336381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.336268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.336381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.336290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.336381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.336306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.336381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.336373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd97812e-9629-4da1-a7e5-dccfcb749b45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.338319 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.338300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.338538 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.338522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.343318 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.343292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd97812e-9629-4da1-a7e5-dccfcb749b45-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.343485 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.343438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j77t\" (UniqueName: \"kubernetes.io/projected/fd97812e-9629-4da1-a7e5-dccfcb749b45-kube-api-access-9j77t\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5j9c8\" (UID: \"fd97812e-9629-4da1-a7e5-dccfcb749b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.470293 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.470196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:09.745321 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.745270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"]
Sep 12 02:48:09.747623 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:09.747601    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd97812e_9629_4da1_a7e5_dccfcb749b45.slice/crio-a0d1685a936a9c48dcc51c23d46fd97f0ae233280f2feb9c9a9ae8bd69ba3325 WatchSource:0}: Error finding container a0d1685a936a9c48dcc51c23d46fd97f0ae233280f2feb9c9a9ae8bd69ba3325: Status 404 returned error can't find the container with id a0d1685a936a9c48dcc51c23d46fd97f0ae233280f2feb9c9a9ae8bd69ba3325
Sep 12 02:48:09.808544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:09.808508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" event={"ID":"fd97812e-9629-4da1-a7e5-dccfcb749b45","Type":"ContainerStarted","Data":"a0d1685a936a9c48dcc51c23d46fd97f0ae233280f2feb9c9a9ae8bd69ba3325"}
Sep 12 02:48:12.856651 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:12.855102    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:48:12.856651 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:12.855199    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:48:12.856651 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:12.855237    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:48:13.828705 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:13.828674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" event={"ID":"fd97812e-9629-4da1-a7e5-dccfcb749b45","Type":"ContainerStarted","Data":"c030f7031429df058382878dc1e92bd633ceeb92f72a5866d9c5f1ceb3ac5206"}
Sep 12 02:48:13.850234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:13.850190    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" podStartSLOduration=1.745383979 podStartE2EDuration="4.850176313s" totalImagesPullingTime="3.104792334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:13.84925526 +0000 UTC m=+712.132168841" watchObservedRunningTime="2026-09-12 02:48:13.850176313 +0000 UTC m=+712.133089894"
Sep 12 02:48:14.471424 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:14.471391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:14.476079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:14.476056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:14.833095 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:14.833013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:14.833910 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:14.833893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:48:25.863660 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.863615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:25.867009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.866982    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:25.871692 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.871672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:25.874244 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.874225    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 12 02:48:25.879243 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.879219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:25.962365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.962332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:25.964875 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.964843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:25.966960 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.966938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0978665-25b4-4198-bdd8-4b2f833ea8a3-config-file\") pod \"limitador-limitador-8588749977-n6bsf\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") " pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:25.967063 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:25.966968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v457n\" (UniqueName: \"kubernetes.io/projected/c0978665-25b4-4198-bdd8-4b2f833ea8a3-kube-api-access-v457n\") pod \"limitador-limitador-8588749977-n6bsf\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") " pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:26.067734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.067686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:26.067922 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.067776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0978665-25b4-4198-bdd8-4b2f833ea8a3-config-file\") pod \"limitador-limitador-8588749977-n6bsf\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") " pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:26.067922 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.067822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v457n\" (UniqueName: \"kubernetes.io/projected/c0978665-25b4-4198-bdd8-4b2f833ea8a3-kube-api-access-v457n\") pod \"limitador-limitador-8588749977-n6bsf\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") " pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:26.068718 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.068695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0978665-25b4-4198-bdd8-4b2f833ea8a3-config-file\") pod \"limitador-limitador-8588749977-n6bsf\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") " pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:26.078022 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.077987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v457n\" (UniqueName: \"kubernetes.io/projected/c0978665-25b4-4198-bdd8-4b2f833ea8a3-kube-api-access-v457n\") pod \"limitador-limitador-8588749977-n6bsf\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") " pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:26.182170 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.182073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:26.454541 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.454438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:26.457687 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:26.457660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0978665_25b4_4198_bdd8_4b2f833ea8a3.slice/crio-1ebeb00c579b4091d7f528915247b066ac7866fb05f1a965b466f7a5e6e20b25 WatchSource:0}: Error finding container 1ebeb00c579b4091d7f528915247b066ac7866fb05f1a965b466f7a5e6e20b25: Status 404 returned error can't find the container with id 1ebeb00c579b4091d7f528915247b066ac7866fb05f1a965b466f7a5e6e20b25
Sep 12 02:48:26.652539 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.652507    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-t48lp"]
Sep 12 02:48:26.656832 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.656815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:26.659335 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.659315    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bgmr\""
Sep 12 02:48:26.662497 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.662476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-t48lp"]
Sep 12 02:48:26.687869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.687844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:26.776558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.776469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdrf\" (UniqueName: \"kubernetes.io/projected/babcb582-ac6c-4461-a95f-696576b6852d-kube-api-access-hsdrf\") pod \"authorino-7bcfd8f9cd-t48lp\" (UID: \"babcb582-ac6c-4461-a95f-696576b6852d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:26.787390 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.787345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:26.877772 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.877732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdrf\" (UniqueName: \"kubernetes.io/projected/babcb582-ac6c-4461-a95f-696576b6852d-kube-api-access-hsdrf\") pod \"authorino-7bcfd8f9cd-t48lp\" (UID: \"babcb582-ac6c-4461-a95f-696576b6852d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:26.887415 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.887388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdrf\" (UniqueName: \"kubernetes.io/projected/babcb582-ac6c-4461-a95f-696576b6852d-kube-api-access-hsdrf\") pod \"authorino-7bcfd8f9cd-t48lp\" (UID: \"babcb582-ac6c-4461-a95f-696576b6852d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:26.887920 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.887889    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-m622q"]
Sep 12 02:48:26.891298 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.891276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf" event={"ID":"c0978665-25b4-4198-bdd8-4b2f833ea8a3","Type":"ContainerStarted","Data":"1ebeb00c579b4091d7f528915247b066ac7866fb05f1a965b466f7a5e6e20b25"}
Sep 12 02:48:26.891409 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.891360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:26.896645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.896622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-m622q"]
Sep 12 02:48:26.967898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.967863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:26.979159 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:26.979133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdj4\" (UniqueName: \"kubernetes.io/projected/1068fab5-0f13-46ac-ae24-74805464bf27-kube-api-access-nhdj4\") pod \"authorino-59c974bd48-m622q\" (UID: \"1068fab5-0f13-46ac-ae24-74805464bf27\") " pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:27.080572 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.080474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdj4\" (UniqueName: \"kubernetes.io/projected/1068fab5-0f13-46ac-ae24-74805464bf27-kube-api-access-nhdj4\") pod \"authorino-59c974bd48-m622q\" (UID: \"1068fab5-0f13-46ac-ae24-74805464bf27\") " pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:27.088543 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.088516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdj4\" (UniqueName: \"kubernetes.io/projected/1068fab5-0f13-46ac-ae24-74805464bf27-kube-api-access-nhdj4\") pod \"authorino-59c974bd48-m622q\" (UID: \"1068fab5-0f13-46ac-ae24-74805464bf27\") " pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:27.203630 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.203593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:27.237011 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.236983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-t48lp"]
Sep 12 02:48:27.238351 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:27.238325    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbabcb582_ac6c_4461_a95f_696576b6852d.slice/crio-5dd2f62508a332406e46c83f83eb238ae718e7b51a3bb6b76e6f306cab7ca77f WatchSource:0}: Error finding container 5dd2f62508a332406e46c83f83eb238ae718e7b51a3bb6b76e6f306cab7ca77f: Status 404 returned error can't find the container with id 5dd2f62508a332406e46c83f83eb238ae718e7b51a3bb6b76e6f306cab7ca77f
Sep 12 02:48:27.477408 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.477379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-m622q"]
Sep 12 02:48:27.644799 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:27.644768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1068fab5_0f13_46ac_ae24_74805464bf27.slice/crio-223f20c0702b64c077b44f9e341459a8f3344f8bb68e60dd6f936d73e23165fb WatchSource:0}: Error finding container 223f20c0702b64c077b44f9e341459a8f3344f8bb68e60dd6f936d73e23165fb: Status 404 returned error can't find the container with id 223f20c0702b64c077b44f9e341459a8f3344f8bb68e60dd6f936d73e23165fb
Sep 12 02:48:27.898496 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.898435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp" event={"ID":"babcb582-ac6c-4461-a95f-696576b6852d","Type":"ContainerStarted","Data":"5dd2f62508a332406e46c83f83eb238ae718e7b51a3bb6b76e6f306cab7ca77f"}
Sep 12 02:48:27.899734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.899706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-m622q" event={"ID":"1068fab5-0f13-46ac-ae24-74805464bf27","Type":"ContainerStarted","Data":"223f20c0702b64c077b44f9e341459a8f3344f8bb68e60dd6f936d73e23165fb"}
Sep 12 02:48:27.901182 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.901161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf" event={"ID":"c0978665-25b4-4198-bdd8-4b2f833ea8a3","Type":"ContainerStarted","Data":"accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45"}
Sep 12 02:48:27.901329 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.901313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:27.917575 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:27.917530    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf" podStartSLOduration=1.677928928 podStartE2EDuration="2.917515651s" totalImagesPullingTime="1.239586723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:27.916593607 +0000 UTC m=+726.199507189" watchObservedRunningTime="2026-09-12 02:48:27.917515651 +0000 UTC m=+726.200429231"
Sep 12 02:48:29.912980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:29.912939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-m622q" event={"ID":"1068fab5-0f13-46ac-ae24-74805464bf27","Type":"ContainerStarted","Data":"273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528"}
Sep 12 02:48:29.914524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:29.914491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp" event={"ID":"babcb582-ac6c-4461-a95f-696576b6852d","Type":"ContainerStarted","Data":"f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6"}
Sep 12 02:48:29.928344 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:29.928290    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-m622q" podStartSLOduration=2.226493423 podStartE2EDuration="3.928274098s" totalImagesPullingTime="1.701780675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:29.927662336 +0000 UTC m=+728.210575916" watchObservedRunningTime="2026-09-12 02:48:29.928274098 +0000 UTC m=+728.211187678"
Sep 12 02:48:29.941631 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:29.941571    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp" podStartSLOduration=1.838193457 podStartE2EDuration="3.941549702s" totalImagesPullingTime="2.103356245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:29.940460553 +0000 UTC m=+728.223374134" watchObservedRunningTime="2026-09-12 02:48:29.941549702 +0000 UTC m=+728.224463285"
Sep 12 02:48:29.965040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:29.965006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-t48lp"]
Sep 12 02:48:31.922894 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:31.922859    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp" podUID="babcb582-ac6c-4461-a95f-696576b6852d" containerName="authorino" containerID="cri-o://f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6" gracePeriod=30
Sep 12 02:48:32.161298 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.161275    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:32.226829 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.226744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsdrf\" (UniqueName: \"kubernetes.io/projected/babcb582-ac6c-4461-a95f-696576b6852d-kube-api-access-hsdrf\") pod \"babcb582-ac6c-4461-a95f-696576b6852d\" (UID: \"babcb582-ac6c-4461-a95f-696576b6852d\") "
Sep 12 02:48:32.229018 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.228986    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babcb582-ac6c-4461-a95f-696576b6852d-kube-api-access-hsdrf" (OuterVolumeSpecName: "kube-api-access-hsdrf") pod "babcb582-ac6c-4461-a95f-696576b6852d" (UID: "babcb582-ac6c-4461-a95f-696576b6852d"). InnerVolumeSpecName "kube-api-access-hsdrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:48:32.327291 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.327265    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsdrf\" (UniqueName: \"kubernetes.io/projected/babcb582-ac6c-4461-a95f-696576b6852d-kube-api-access-hsdrf\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:32.927783 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.927741    2579 generic.go:356] "Generic (PLEG): container finished" podID="babcb582-ac6c-4461-a95f-696576b6852d" containerID="f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6" exitCode=0
Sep 12 02:48:32.928173 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.927805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp"
Sep 12 02:48:32.928173 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.927832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp" event={"ID":"babcb582-ac6c-4461-a95f-696576b6852d","Type":"ContainerDied","Data":"f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6"}
Sep 12 02:48:32.928173 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.927874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-t48lp" event={"ID":"babcb582-ac6c-4461-a95f-696576b6852d","Type":"ContainerDied","Data":"5dd2f62508a332406e46c83f83eb238ae718e7b51a3bb6b76e6f306cab7ca77f"}
Sep 12 02:48:32.928173 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.927890    2579 scope.go:117] "RemoveContainer" containerID="f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6"
Sep 12 02:48:32.937196 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.937174    2579 scope.go:117] "RemoveContainer" containerID="f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6"
Sep 12 02:48:32.938379 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:48:32.938346    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6\": container with ID starting with f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6 not found: ID does not exist" containerID="f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6"
Sep 12 02:48:32.938505 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.938386    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6"} err="failed to get container status \"f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6\": rpc error: code = NotFound desc = could not find container \"f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6\": container with ID starting with f6b167ae5b24d400095bce693515972f824820d75c384420c38ed74accb58ce6 not found: ID does not exist"
Sep 12 02:48:32.943251 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.943228    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-t48lp"]
Sep 12 02:48:32.946763 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:32.946743    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-t48lp"]
Sep 12 02:48:34.244639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:34.244606    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babcb582-ac6c-4461-a95f-696576b6852d" path="/var/lib/kubelet/pods/babcb582-ac6c-4461-a95f-696576b6852d/volumes"
Sep 12 02:48:38.908438 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:38.908413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:39.405400 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.405366    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:39.405625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.405602    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf" podUID="c0978665-25b4-4198-bdd8-4b2f833ea8a3" containerName="limitador" containerID="cri-o://accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45" gracePeriod=30
Sep 12 02:48:39.948793 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.948767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:39.965682 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.965616    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0978665-25b4-4198-bdd8-4b2f833ea8a3" containerID="accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45" exitCode=0
Sep 12 02:48:39.965682 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.965659    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf"
Sep 12 02:48:39.965855 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.965734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf" event={"ID":"c0978665-25b4-4198-bdd8-4b2f833ea8a3","Type":"ContainerDied","Data":"accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45"}
Sep 12 02:48:39.965855 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.965806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-n6bsf" event={"ID":"c0978665-25b4-4198-bdd8-4b2f833ea8a3","Type":"ContainerDied","Data":"1ebeb00c579b4091d7f528915247b066ac7866fb05f1a965b466f7a5e6e20b25"}
Sep 12 02:48:39.965855 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.965840    2579 scope.go:117] "RemoveContainer" containerID="accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45"
Sep 12 02:48:39.994590 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.994572    2579 scope.go:117] "RemoveContainer" containerID="accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45"
Sep 12 02:48:39.994887 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:48:39.994867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45\": container with ID starting with accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45 not found: ID does not exist" containerID="accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45"
Sep 12 02:48:39.994939 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:39.994898    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45"} err="failed to get container status \"accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45\": rpc error: code = NotFound desc = could not find container \"accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45\": container with ID starting with accf08de7cf8139706296e9b2c72d3aaf2bf0b91ada83ef77bc1625611250f45 not found: ID does not exist"
Sep 12 02:48:40.089886 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.089862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v457n\" (UniqueName: \"kubernetes.io/projected/c0978665-25b4-4198-bdd8-4b2f833ea8a3-kube-api-access-v457n\") pod \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") "
Sep 12 02:48:40.090011 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.089937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0978665-25b4-4198-bdd8-4b2f833ea8a3-config-file\") pod \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\" (UID: \"c0978665-25b4-4198-bdd8-4b2f833ea8a3\") "
Sep 12 02:48:40.090295 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.090273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0978665-25b4-4198-bdd8-4b2f833ea8a3-config-file" (OuterVolumeSpecName: "config-file") pod "c0978665-25b4-4198-bdd8-4b2f833ea8a3" (UID: "c0978665-25b4-4198-bdd8-4b2f833ea8a3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:48:40.092089 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.092064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0978665-25b4-4198-bdd8-4b2f833ea8a3-kube-api-access-v457n" (OuterVolumeSpecName: "kube-api-access-v457n") pod "c0978665-25b4-4198-bdd8-4b2f833ea8a3" (UID: "c0978665-25b4-4198-bdd8-4b2f833ea8a3"). InnerVolumeSpecName "kube-api-access-v457n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:48:40.190587 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.190559    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v457n\" (UniqueName: \"kubernetes.io/projected/c0978665-25b4-4198-bdd8-4b2f833ea8a3-kube-api-access-v457n\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:40.190587 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.190584    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0978665-25b4-4198-bdd8-4b2f833ea8a3-config-file\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:40.280621 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.280593    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:40.282560 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:40.282539    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-n6bsf"]
Sep 12 02:48:41.278163 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.278128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:41.375183 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.375156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:41.748585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.748555    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"]
Sep 12 02:48:41.748930 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.748914    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0978665-25b4-4198-bdd8-4b2f833ea8a3" containerName="limitador"
Sep 12 02:48:41.749030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.748933    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0978665-25b4-4198-bdd8-4b2f833ea8a3" containerName="limitador"
Sep 12 02:48:41.749030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.748960    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babcb582-ac6c-4461-a95f-696576b6852d" containerName="authorino"
Sep 12 02:48:41.749030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.748969    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="babcb582-ac6c-4461-a95f-696576b6852d" containerName="authorino"
Sep 12 02:48:41.749169 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.749047    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0978665-25b4-4198-bdd8-4b2f833ea8a3" containerName="limitador"
Sep 12 02:48:41.749169 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.749062    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="babcb582-ac6c-4461-a95f-696576b6852d" containerName="authorino"
Sep 12 02:48:41.753149 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.753128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:41.756159 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.756136    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 02:48:41.756277 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.756141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 12 02:48:41.756365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.756347    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kvxg6\""
Sep 12 02:48:41.757438 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.757420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 02:48:41.759365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.759341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"]
Sep 12 02:48:41.773903 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.773880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:41.902980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.902954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5644156-4463-4095-bb31-ece4310f815f-data\") pod \"postgres-69fb4594cf-stjlq\" (UID: \"c5644156-4463-4095-bb31-ece4310f815f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:41.903095 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:41.902984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgxw\" (UniqueName: \"kubernetes.io/projected/c5644156-4463-4095-bb31-ece4310f815f-kube-api-access-7jgxw\") pod \"postgres-69fb4594cf-stjlq\" (UID: \"c5644156-4463-4095-bb31-ece4310f815f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:42.004064 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.003978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5644156-4463-4095-bb31-ece4310f815f-data\") pod \"postgres-69fb4594cf-stjlq\" (UID: \"c5644156-4463-4095-bb31-ece4310f815f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:42.004064 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.004018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgxw\" (UniqueName: \"kubernetes.io/projected/c5644156-4463-4095-bb31-ece4310f815f-kube-api-access-7jgxw\") pod \"postgres-69fb4594cf-stjlq\" (UID: \"c5644156-4463-4095-bb31-ece4310f815f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:42.004341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.004320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c5644156-4463-4095-bb31-ece4310f815f-data\") pod \"postgres-69fb4594cf-stjlq\" (UID: \"c5644156-4463-4095-bb31-ece4310f815f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:42.012959 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.012930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgxw\" (UniqueName: \"kubernetes.io/projected/c5644156-4463-4095-bb31-ece4310f815f-kube-api-access-7jgxw\") pod \"postgres-69fb4594cf-stjlq\" (UID: \"c5644156-4463-4095-bb31-ece4310f815f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:42.064162 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.064141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:42.245383 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.245353    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0978665-25b4-4198-bdd8-4b2f833ea8a3" path="/var/lib/kubelet/pods/c0978665-25b4-4198-bdd8-4b2f833ea8a3/volumes"
Sep 12 02:48:42.332272 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.332132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"]
Sep 12 02:48:42.334699 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:42.334672    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5644156_4463_4095_bb31_ece4310f815f.slice/crio-336f9744be2e9fb0ed8e7b030909f7d138e80a5c4cecbd23c160d1bd4ac35362 WatchSource:0}: Error finding container 336f9744be2e9fb0ed8e7b030909f7d138e80a5c4cecbd23c160d1bd4ac35362: Status 404 returned error can't find the container with id 336f9744be2e9fb0ed8e7b030909f7d138e80a5c4cecbd23c160d1bd4ac35362
Sep 12 02:48:42.979909 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:42.979879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq" event={"ID":"c5644156-4463-4095-bb31-ece4310f815f","Type":"ContainerStarted","Data":"336f9744be2e9fb0ed8e7b030909f7d138e80a5c4cecbd23c160d1bd4ac35362"}
Sep 12 02:48:44.991595 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:44.991551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq" event={"ID":"c5644156-4463-4095-bb31-ece4310f815f","Type":"ContainerStarted","Data":"0f063cc64d45d378f720da875bbc2d80585c9ac97cf5b925a2f35f91e18ec94c"}
Sep 12 02:48:44.992016 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:44.991657    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:45.007475 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:45.007415    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq" podStartSLOduration=1.9105365619999999 podStartE2EDuration="4.007402318s" totalImagesPullingTime="2.096865756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:45.005984749 +0000 UTC m=+743.288898329" watchObservedRunningTime="2026-09-12 02:48:45.007402318 +0000 UTC m=+743.290315899"
Sep 12 02:48:51.023921 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.023894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-stjlq"
Sep 12 02:48:51.341363 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.341278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:51.558458 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.558414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-2zt2n"]
Sep 12 02:48:51.561653 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.561632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:51.568978 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.568958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-2zt2n"]
Sep 12 02:48:51.591122 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.591096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:51.634199 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.634170    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2zt2n"]
Sep 12 02:48:51.634387 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:48:51.634370    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p2bb5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-2zt2n" podUID="7822b6e1-11c6-4886-befb-338ffeada459"
Sep 12 02:48:51.681529 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.681498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2bb5\" (UniqueName: \"kubernetes.io/projected/7822b6e1-11c6-4886-befb-338ffeada459-kube-api-access-p2bb5\") pod \"authorino-557f876f69-2zt2n\" (UID: \"7822b6e1-11c6-4886-befb-338ffeada459\") " pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:51.782900 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.782868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2bb5\" (UniqueName: \"kubernetes.io/projected/7822b6e1-11c6-4886-befb-338ffeada459-kube-api-access-p2bb5\") pod \"authorino-557f876f69-2zt2n\" (UID: \"7822b6e1-11c6-4886-befb-338ffeada459\") " pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:51.790962 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.790932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2bb5\" (UniqueName: \"kubernetes.io/projected/7822b6e1-11c6-4886-befb-338ffeada459-kube-api-access-p2bb5\") pod \"authorino-557f876f69-2zt2n\" (UID: \"7822b6e1-11c6-4886-befb-338ffeada459\") " pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:51.825256 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.825229    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b67b44488-s85qd"]
Sep 12 02:48:51.828208 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.828192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:51.830713 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.830697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 12 02:48:51.837486 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.837468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b67b44488-s85qd"]
Sep 12 02:48:51.985425 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.985335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c6a227-9d1a-4f3d-9018-35b9862ecb32-tls-cert\") pod \"authorino-b67b44488-s85qd\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") " pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:51.985425 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:51.985388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpth\" (UniqueName: \"kubernetes.io/projected/39c6a227-9d1a-4f3d-9018-35b9862ecb32-kube-api-access-5vpth\") pod \"authorino-b67b44488-s85qd\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") " pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:52.020914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.020879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:52.027179 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.027158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:52.086437 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.086412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c6a227-9d1a-4f3d-9018-35b9862ecb32-tls-cert\") pod \"authorino-b67b44488-s85qd\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") " pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:52.086586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.086468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpth\" (UniqueName: \"kubernetes.io/projected/39c6a227-9d1a-4f3d-9018-35b9862ecb32-kube-api-access-5vpth\") pod \"authorino-b67b44488-s85qd\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") " pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:52.088959 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.088939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c6a227-9d1a-4f3d-9018-35b9862ecb32-tls-cert\") pod \"authorino-b67b44488-s85qd\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") " pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:52.094282 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.094232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpth\" (UniqueName: \"kubernetes.io/projected/39c6a227-9d1a-4f3d-9018-35b9862ecb32-kube-api-access-5vpth\") pod \"authorino-b67b44488-s85qd\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") " pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:52.136852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.136837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:48:52.187099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.186904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2bb5\" (UniqueName: \"kubernetes.io/projected/7822b6e1-11c6-4886-befb-338ffeada459-kube-api-access-p2bb5\") pod \"7822b6e1-11c6-4886-befb-338ffeada459\" (UID: \"7822b6e1-11c6-4886-befb-338ffeada459\") "
Sep 12 02:48:52.190891 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.190866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7822b6e1-11c6-4886-befb-338ffeada459-kube-api-access-p2bb5" (OuterVolumeSpecName: "kube-api-access-p2bb5") pod "7822b6e1-11c6-4886-befb-338ffeada459" (UID: "7822b6e1-11c6-4886-befb-338ffeada459"). InnerVolumeSpecName "kube-api-access-p2bb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:48:52.287987 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.287963    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2bb5\" (UniqueName: \"kubernetes.io/projected/7822b6e1-11c6-4886-befb-338ffeada459-kube-api-access-p2bb5\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:52.402174 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:52.402015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b67b44488-s85qd"]
Sep 12 02:48:52.404614 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:52.404592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39c6a227_9d1a_4f3d_9018_35b9862ecb32.slice/crio-d5024cd970ee818653e49cc2e86f1df97db5d4f56af876d4c712977166e3503a WatchSource:0}: Error finding container d5024cd970ee818653e49cc2e86f1df97db5d4f56af876d4c712977166e3503a: Status 404 returned error can't find the container with id d5024cd970ee818653e49cc2e86f1df97db5d4f56af876d4c712977166e3503a
Sep 12 02:48:53.026172 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.026148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2zt2n"
Sep 12 02:48:53.026338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.026150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b67b44488-s85qd" event={"ID":"39c6a227-9d1a-4f3d-9018-35b9862ecb32","Type":"ContainerStarted","Data":"baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e"}
Sep 12 02:48:53.026338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.026250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b67b44488-s85qd" event={"ID":"39c6a227-9d1a-4f3d-9018-35b9862ecb32","Type":"ContainerStarted","Data":"d5024cd970ee818653e49cc2e86f1df97db5d4f56af876d4c712977166e3503a"}
Sep 12 02:48:53.045663 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.045577    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b67b44488-s85qd" podStartSLOduration=1.6727247809999999 podStartE2EDuration="2.045564533s" totalImagesPullingTime="372.839752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:48:53.04419127 +0000 UTC m=+751.327104853" watchObservedRunningTime="2026-09-12 02:48:53.045564533 +0000 UTC m=+751.328478113"
Sep 12 02:48:53.066816 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.066793    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2zt2n"]
Sep 12 02:48:53.074424 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.074394    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-m622q"]
Sep 12 02:48:53.074651 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.074616    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-m622q" podUID="1068fab5-0f13-46ac-ae24-74805464bf27" containerName="authorino" containerID="cri-o://273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528" gracePeriod=30
Sep 12 02:48:53.076390 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.076365    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-2zt2n"]
Sep 12 02:48:53.325346 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.325323    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:53.498020 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.497986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhdj4\" (UniqueName: \"kubernetes.io/projected/1068fab5-0f13-46ac-ae24-74805464bf27-kube-api-access-nhdj4\") pod \"1068fab5-0f13-46ac-ae24-74805464bf27\" (UID: \"1068fab5-0f13-46ac-ae24-74805464bf27\") "
Sep 12 02:48:53.500220 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.500191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1068fab5-0f13-46ac-ae24-74805464bf27-kube-api-access-nhdj4" (OuterVolumeSpecName: "kube-api-access-nhdj4") pod "1068fab5-0f13-46ac-ae24-74805464bf27" (UID: "1068fab5-0f13-46ac-ae24-74805464bf27"). InnerVolumeSpecName "kube-api-access-nhdj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:48:53.599284 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:53.599200    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdj4\" (UniqueName: \"kubernetes.io/projected/1068fab5-0f13-46ac-ae24-74805464bf27-kube-api-access-nhdj4\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:48:54.032153 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.032109    2579 generic.go:356] "Generic (PLEG): container finished" podID="1068fab5-0f13-46ac-ae24-74805464bf27" containerID="273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528" exitCode=0
Sep 12 02:48:54.032153 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.032159    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-m622q"
Sep 12 02:48:54.032393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.032155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-m622q" event={"ID":"1068fab5-0f13-46ac-ae24-74805464bf27","Type":"ContainerDied","Data":"273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528"}
Sep 12 02:48:54.032393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.032204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-m622q" event={"ID":"1068fab5-0f13-46ac-ae24-74805464bf27","Type":"ContainerDied","Data":"223f20c0702b64c077b44f9e341459a8f3344f8bb68e60dd6f936d73e23165fb"}
Sep 12 02:48:54.032393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.032223    2579 scope.go:117] "RemoveContainer" containerID="273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528"
Sep 12 02:48:54.040941 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.040923    2579 scope.go:117] "RemoveContainer" containerID="273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528"
Sep 12 02:48:54.041294 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:48:54.041276    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528\": container with ID starting with 273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528 not found: ID does not exist" containerID="273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528"
Sep 12 02:48:54.041350 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.041300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528"} err="failed to get container status \"273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528\": rpc error: code = NotFound desc = could not find container \"273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528\": container with ID starting with 273a76fe6d7128de31a8d22e357e6bfb6b33cecb46903f1edce9dcb776b01528 not found: ID does not exist"
Sep 12 02:48:54.052836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.052814    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-m622q"]
Sep 12 02:48:54.055151 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.055129    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-m622q"]
Sep 12 02:48:54.245318 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.245285    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1068fab5-0f13-46ac-ae24-74805464bf27" path="/var/lib/kubelet/pods/1068fab5-0f13-46ac-ae24-74805464bf27/volumes"
Sep 12 02:48:54.245622 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:54.245609    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7822b6e1-11c6-4886-befb-338ffeada459" path="/var/lib/kubelet/pods/7822b6e1-11c6-4886-befb-338ffeada459/volumes"
Sep 12 02:48:58.009359 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.009326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:58.032439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.032400    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fdc96c449-hf577"]
Sep 12 02:48:58.032727 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.032714    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1068fab5-0f13-46ac-ae24-74805464bf27" containerName="authorino"
Sep 12 02:48:58.032774 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.032728    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1068fab5-0f13-46ac-ae24-74805464bf27" containerName="authorino"
Sep 12 02:48:58.032814 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.032786    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1068fab5-0f13-46ac-ae24-74805464bf27" containerName="authorino"
Sep 12 02:48:58.038072 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.038048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.041573 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.041550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 02:48:58.041573 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.041562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fs74v\""
Sep 12 02:48:58.041802 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.041654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 02:48:58.042726 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.042707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 02:48:58.048339 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.048317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fdc96c449-hf577"]
Sep 12 02:48:58.103967 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.103924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:58.136116 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.136091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-webhook-cert\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.136261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.136125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-metrics-certs\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.136261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.136147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f52g\" (UniqueName: \"kubernetes.io/projected/034d6b10-246e-409c-96c7-203092fa28c7-kube-api-access-7f52g\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.204747 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.204719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:58.210802 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.210780    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5848789547-brjgr"]
Sep 12 02:48:58.214065 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.214048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.223905 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.223884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5848789547-brjgr"]
Sep 12 02:48:58.237245 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.237223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-metrics-certs\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.237338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.237256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f52g\" (UniqueName: \"kubernetes.io/projected/034d6b10-246e-409c-96c7-203092fa28c7-kube-api-access-7f52g\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.237338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.237276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-webhook-cert\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.237338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.237314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvssk\" (UniqueName: \"kubernetes.io/projected/2003500d-8b2d-433c-83f6-ffd91f67c6f9-kube-api-access-gvssk\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.237524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.237345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-metrics-certs\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.237524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.237399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-webhook-cert\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.239813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.239792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-metrics-certs\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.239985 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.239966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-webhook-cert\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.245079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.245051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f52g\" (UniqueName: \"kubernetes.io/projected/034d6b10-246e-409c-96c7-203092fa28c7-kube-api-access-7f52g\") pod \"maas-controller-5fdc96c449-hf577\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") " pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.262573 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.262518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:48:58.338016 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.337982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-webhook-cert\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.338160 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.338032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvssk\" (UniqueName: \"kubernetes.io/projected/2003500d-8b2d-433c-83f6-ffd91f67c6f9-kube-api-access-gvssk\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.338209 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.338149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-metrics-certs\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.340590 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.340571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-metrics-certs\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.340689 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.340595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-webhook-cert\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.345624 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.345601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvssk\" (UniqueName: \"kubernetes.io/projected/2003500d-8b2d-433c-83f6-ffd91f67c6f9-kube-api-access-gvssk\") pod \"maas-controller-5848789547-brjgr\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") " pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.351307 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.351287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:48:58.524425 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.524353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:48:58.622089 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.622062    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fdc96c449-hf577"]
Sep 12 02:48:58.623600 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:58.623575    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034d6b10_246e_409c_96c7_203092fa28c7.slice/crio-0dc3a3eb131c199943ac6604675c693f7ecd085e1f83825924e72b8596e11463 WatchSource:0}: Error finding container 0dc3a3eb131c199943ac6604675c693f7ecd085e1f83825924e72b8596e11463: Status 404 returned error can't find the container with id 0dc3a3eb131c199943ac6604675c693f7ecd085e1f83825924e72b8596e11463
Sep 12 02:48:58.793046 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:58.793021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5848789547-brjgr"]
Sep 12 02:48:58.795175 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:48:58.795151    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2003500d_8b2d_433c_83f6_ffd91f67c6f9.slice/crio-7a0a2f093f749851c65d8a2b0fbba62c1c797f62137df5b21ba8f5a7a7b6ea66 WatchSource:0}: Error finding container 7a0a2f093f749851c65d8a2b0fbba62c1c797f62137df5b21ba8f5a7a7b6ea66: Status 404 returned error can't find the container with id 7a0a2f093f749851c65d8a2b0fbba62c1c797f62137df5b21ba8f5a7a7b6ea66
Sep 12 02:48:59.055549 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:59.055467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-brjgr" event={"ID":"2003500d-8b2d-433c-83f6-ffd91f67c6f9","Type":"ContainerStarted","Data":"7a0a2f093f749851c65d8a2b0fbba62c1c797f62137df5b21ba8f5a7a7b6ea66"}
Sep 12 02:48:59.056640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:48:59.056619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc96c449-hf577" event={"ID":"034d6b10-246e-409c-96c7-203092fa28c7","Type":"ContainerStarted","Data":"0dc3a3eb131c199943ac6604675c693f7ecd085e1f83825924e72b8596e11463"}
Sep 12 02:49:01.454114 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:01.454086    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:01.556522 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:01.556419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:02.073287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:02.073246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-brjgr" event={"ID":"2003500d-8b2d-433c-83f6-ffd91f67c6f9","Type":"ContainerStarted","Data":"cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857"}
Sep 12 02:49:02.073508 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:02.073361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:49:02.075393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:02.075369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc96c449-hf577" event={"ID":"034d6b10-246e-409c-96c7-203092fa28c7","Type":"ContainerStarted","Data":"de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b"}
Sep 12 02:49:02.075545 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:02.075500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:49:02.092267 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:02.092223    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5848789547-brjgr" podStartSLOduration=1.63701811 podStartE2EDuration="4.092207865s" totalImagesPullingTime="2.455189755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:49:02.091628044 +0000 UTC m=+760.374541630" watchObservedRunningTime="2026-09-12 02:49:02.092207865 +0000 UTC m=+760.375121446"
Sep 12 02:49:02.112279 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:02.112224    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fdc96c449-hf577" podStartSLOduration=1.494526128 podStartE2EDuration="4.112206995s" totalImagesPullingTime="2.617680867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:49:02.111002437 +0000 UTC m=+760.393916014" watchObservedRunningTime="2026-09-12 02:49:02.112206995 +0000 UTC m=+760.395120577"
Sep 12 02:49:13.081266 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.081233    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 02:49:13.081654 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.081504    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:49:13.140559 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.140532    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fdc96c449-hf577"]
Sep 12 02:49:13.140771 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.140717    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fdc96c449-hf577" podUID="034d6b10-246e-409c-96c7-203092fa28c7" containerName="manager" containerID="cri-o://de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b" gracePeriod=30
Sep 12 02:49:13.379995 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.379972    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:49:13.446708 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.446679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f52g\" (UniqueName: \"kubernetes.io/projected/034d6b10-246e-409c-96c7-203092fa28c7-kube-api-access-7f52g\") pod \"034d6b10-246e-409c-96c7-203092fa28c7\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") "
Sep 12 02:49:13.446869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.446727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-webhook-cert\") pod \"034d6b10-246e-409c-96c7-203092fa28c7\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") "
Sep 12 02:49:13.446869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.446771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-metrics-certs\") pod \"034d6b10-246e-409c-96c7-203092fa28c7\" (UID: \"034d6b10-246e-409c-96c7-203092fa28c7\") "
Sep 12 02:49:13.449012 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.448969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "034d6b10-246e-409c-96c7-203092fa28c7" (UID: "034d6b10-246e-409c-96c7-203092fa28c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:49:13.449136 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.449026    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034d6b10-246e-409c-96c7-203092fa28c7-kube-api-access-7f52g" (OuterVolumeSpecName: "kube-api-access-7f52g") pod "034d6b10-246e-409c-96c7-203092fa28c7" (UID: "034d6b10-246e-409c-96c7-203092fa28c7"). InnerVolumeSpecName "kube-api-access-7f52g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:49:13.449136 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.449033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "034d6b10-246e-409c-96c7-203092fa28c7" (UID: "034d6b10-246e-409c-96c7-203092fa28c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:49:13.547625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.547597    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:49:13.547625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.547620    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f52g\" (UniqueName: \"kubernetes.io/projected/034d6b10-246e-409c-96c7-203092fa28c7-kube-api-access-7f52g\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:49:13.547625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:13.547630    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/034d6b10-246e-409c-96c7-203092fa28c7-webhook-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:49:14.128546 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.128516    2579 generic.go:356] "Generic (PLEG): container finished" podID="034d6b10-246e-409c-96c7-203092fa28c7" containerID="de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b" exitCode=0
Sep 12 02:49:14.128970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.128553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc96c449-hf577" event={"ID":"034d6b10-246e-409c-96c7-203092fa28c7","Type":"ContainerDied","Data":"de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b"}
Sep 12 02:49:14.128970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.128559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fdc96c449-hf577"
Sep 12 02:49:14.128970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.128596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fdc96c449-hf577" event={"ID":"034d6b10-246e-409c-96c7-203092fa28c7","Type":"ContainerDied","Data":"0dc3a3eb131c199943ac6604675c693f7ecd085e1f83825924e72b8596e11463"}
Sep 12 02:49:14.128970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.128615    2579 scope.go:117] "RemoveContainer" containerID="de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b"
Sep 12 02:49:14.137248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.137226    2579 scope.go:117] "RemoveContainer" containerID="de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b"
Sep 12 02:49:14.137555 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:49:14.137538    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b\": container with ID starting with de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b not found: ID does not exist" containerID="de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b"
Sep 12 02:49:14.137626 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.137563    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b"} err="failed to get container status \"de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b\": rpc error: code = NotFound desc = could not find container \"de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b\": container with ID starting with de6b2ee6398e7ee55d2b865b55c4993aaaba95f3f8f4c29509d2572ea7422f5b not found: ID does not exist"
Sep 12 02:49:14.153403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.152154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fdc96c449-hf577"]
Sep 12 02:49:14.156941 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.156915    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fdc96c449-hf577"]
Sep 12 02:49:14.244431 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:14.244398    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="034d6b10-246e-409c-96c7-203092fa28c7" path="/var/lib/kubelet/pods/034d6b10-246e-409c-96c7-203092fa28c7/volumes"
Sep 12 02:49:32.148757 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:32.148718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:32.249474 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:32.249420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:33.640701 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.640663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:33.745096 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.745062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:33.782892 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.782863    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"]
Sep 12 02:49:33.783173 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.783162    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="034d6b10-246e-409c-96c7-203092fa28c7" containerName="manager"
Sep 12 02:49:33.783215 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.783175    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="034d6b10-246e-409c-96c7-203092fa28c7" containerName="manager"
Sep 12 02:49:33.783248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.783229    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="034d6b10-246e-409c-96c7-203092fa28c7" containerName="manager"
Sep 12 02:49:33.791204 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.791186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:33.795412 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.795389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsxrv\""
Sep 12 02:49:33.795638 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.795618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 02:49:33.795722 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.795650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 12 02:49:33.797254 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.797233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"]
Sep 12 02:49:33.802126 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.802100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:33.912708 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.912634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f90316-0fdd-4202-9607-c259e0fa3eb7-openshift-service-ca\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:33.912708 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.912692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gtz\" (UniqueName: \"kubernetes.io/projected/75f90316-0fdd-4202-9607-c259e0fa3eb7-kube-api-access-76gtz\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:33.912902 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.912760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-metrics-certs\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:33.912902 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.912803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-maas-api-tls\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:33.953741 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:33.953712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:34.014063 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.014022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gtz\" (UniqueName: \"kubernetes.io/projected/75f90316-0fdd-4202-9607-c259e0fa3eb7-kube-api-access-76gtz\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.014240 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.014082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-metrics-certs\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.014240 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.014118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-maas-api-tls\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.014240 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.014159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f90316-0fdd-4202-9607-c259e0fa3eb7-openshift-service-ca\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.014922 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.014871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f90316-0fdd-4202-9607-c259e0fa3eb7-openshift-service-ca\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.016883 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.016860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-maas-api-tls\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.017190 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.017173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-metrics-certs\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.021646 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.021623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gtz\" (UniqueName: \"kubernetes.io/projected/75f90316-0fdd-4202-9607-c259e0fa3eb7-kube-api-access-76gtz\") pod \"maas-api-bb8f5d89c-nqwpl\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") " pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.103146 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.103109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:34.178245 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.178173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:34.280037 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.280009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:34.317912 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.317883    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"]
Sep 12 02:49:34.347810 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.347785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"]
Sep 12 02:49:34.347952 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.347847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.350722 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.350675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-v6nz2\""
Sep 12 02:49:34.350879 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.350759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 12 02:49:34.381879 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.381835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"]
Sep 12 02:49:34.384699 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:49:34.384650    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f90316_0fdd_4202_9607_c259e0fa3eb7.slice/crio-221306913e2a009b312b654558e2f09ca8098114c198f951455f4ce6db59a967 WatchSource:0}: Error finding container 221306913e2a009b312b654558e2f09ca8098114c198f951455f4ce6db59a967: Status 404 returned error can't find the container with id 221306913e2a009b312b654558e2f09ca8098114c198f951455f4ce6db59a967
Sep 12 02:49:34.417889 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.417855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2k5\" (UniqueName: \"kubernetes.io/projected/0b574436-4216-4db3-8a6d-f53d83910cf7-kube-api-access-ps2k5\") pod \"payload-processing-5b54b7fbb7-l7qnf\" (UID: \"0b574436-4216-4db3-8a6d-f53d83910cf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.418074 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.417911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b574436-4216-4db3-8a6d-f53d83910cf7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-l7qnf\" (UID: \"0b574436-4216-4db3-8a6d-f53d83910cf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.518673 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.518521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2k5\" (UniqueName: \"kubernetes.io/projected/0b574436-4216-4db3-8a6d-f53d83910cf7-kube-api-access-ps2k5\") pod \"payload-processing-5b54b7fbb7-l7qnf\" (UID: \"0b574436-4216-4db3-8a6d-f53d83910cf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.518673 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.518634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b574436-4216-4db3-8a6d-f53d83910cf7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-l7qnf\" (UID: \"0b574436-4216-4db3-8a6d-f53d83910cf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.519463 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.519428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b574436-4216-4db3-8a6d-f53d83910cf7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-l7qnf\" (UID: \"0b574436-4216-4db3-8a6d-f53d83910cf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.526912 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.526882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2k5\" (UniqueName: \"kubernetes.io/projected/0b574436-4216-4db3-8a6d-f53d83910cf7-kube-api-access-ps2k5\") pod \"payload-processing-5b54b7fbb7-l7qnf\" (UID: \"0b574436-4216-4db3-8a6d-f53d83910cf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.659219 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.659181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:34.933883 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:34.933862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"]
Sep 12 02:49:34.936721 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:49:34.936697    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b574436_4216_4db3_8a6d_f53d83910cf7.slice/crio-f0f8b61e13a9ebe7c3e824a59a464dd4541cb669bb7b29bfc5e757db6279439d WatchSource:0}: Error finding container f0f8b61e13a9ebe7c3e824a59a464dd4541cb669bb7b29bfc5e757db6279439d: Status 404 returned error can't find the container with id f0f8b61e13a9ebe7c3e824a59a464dd4541cb669bb7b29bfc5e757db6279439d
Sep 12 02:49:35.223762 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:35.223678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" event={"ID":"75f90316-0fdd-4202-9607-c259e0fa3eb7","Type":"ContainerStarted","Data":"221306913e2a009b312b654558e2f09ca8098114c198f951455f4ce6db59a967"}
Sep 12 02:49:35.224926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:35.224883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf" event={"ID":"0b574436-4216-4db3-8a6d-f53d83910cf7","Type":"ContainerStarted","Data":"f0f8b61e13a9ebe7c3e824a59a464dd4541cb669bb7b29bfc5e757db6279439d"}
Sep 12 02:49:37.236822 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:37.236741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" event={"ID":"75f90316-0fdd-4202-9607-c259e0fa3eb7","Type":"ContainerStarted","Data":"e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6"}
Sep 12 02:49:37.237283 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:37.236833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:37.238494 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:37.238431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf" event={"ID":"0b574436-4216-4db3-8a6d-f53d83910cf7","Type":"ContainerStarted","Data":"ad9bb65dea26f8162a6aab187fda3ed1af4772438e56ccc4d835891056812cda"}
Sep 12 02:49:37.238611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:37.238540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:37.254392 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:37.254345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" podStartSLOduration=1.732734771 podStartE2EDuration="4.254327839s" totalImagesPullingTime="2.521593068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:49:37.2530866 +0000 UTC m=+795.536000182" watchObservedRunningTime="2026-09-12 02:49:37.254327839 +0000 UTC m=+795.537241422"
Sep 12 02:49:37.271186 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:37.271124    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf" podStartSLOduration=1.299358798 podStartE2EDuration="3.27110515s" totalImagesPullingTime="1.971746352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:49:37.268595232 +0000 UTC m=+795.551508816" watchObservedRunningTime="2026-09-12 02:49:37.27110515 +0000 UTC m=+795.554018735"
Sep 12 02:49:40.661878 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:40.661849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:43.248628 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:43.248546    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 02:49:44.270177 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.270135    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd97812e-9629-4da1-a7e5-dccfcb749b45" containerID="c030f7031429df058382878dc1e92bd633ceeb92f72a5866d9c5f1ceb3ac5206" exitCode=0
Sep 12 02:49:44.270644 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.270233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" event={"ID":"fd97812e-9629-4da1-a7e5-dccfcb749b45","Type":"ContainerDied","Data":"c030f7031429df058382878dc1e92bd633ceeb92f72a5866d9c5f1ceb3ac5206"}
Sep 12 02:49:44.270644 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.270637    2579 scope.go:117] "RemoveContainer" containerID="c030f7031429df058382878dc1e92bd633ceeb92f72a5866d9c5f1ceb3ac5206"
Sep 12 02:49:44.271585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.271559    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:49:44.271653 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.271625    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:49:44.271689 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.271656    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:49:44.279932 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.279908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:44.377372 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.377334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:49:44.833440 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:44.833397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:49:45.280211 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:45.280165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" event={"ID":"fd97812e-9629-4da1-a7e5-dccfcb749b45","Type":"ContainerStarted","Data":"857bafe0981580725ce69569dbe7174c753ac8e2ba5c6ad9b67ca32b76366c71"}
Sep 12 02:49:45.471989 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:45.471943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:49:45.472197 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:45.472112    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5j9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 12 02:49:45.472197 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:45.472157    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" podUID="fd97812e-9629-4da1-a7e5-dccfcb749b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 12 02:49:46.471952 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:46.471507    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5j9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 12 02:49:46.471952 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:46.471572    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" podUID="fd97812e-9629-4da1-a7e5-dccfcb749b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 12 02:49:47.470874 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:47.470836    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5j9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 12 02:49:47.471211 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:47.471178    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" podUID="fd97812e-9629-4da1-a7e5-dccfcb749b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 12 02:49:48.244872 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:48.244847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-l7qnf"
Sep 12 02:49:48.471698 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:48.471653    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5j9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 12 02:49:48.471902 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:48.471737    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" podUID="fd97812e-9629-4da1-a7e5-dccfcb749b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 12 02:49:49.470940 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:49.470897    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5j9c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 12 02:49:49.471502 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:49.470968    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8" podUID="fd97812e-9629-4da1-a7e5-dccfcb749b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 12 02:49:50.475090 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:50.475061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:49:50.475576 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:50.475325    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:49:50.476062 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:50.476047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5j9c8"
Sep 12 02:49:54.757154 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:49:54.757126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:08.116808 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:08.116770    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:08.790321 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:08.790286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:12.206679 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.206649    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"]
Sep 12 02:50:12.213215 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.213192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.215952 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.215931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 02:50:12.217368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.217348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 12 02:50:12.217555 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.217442    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wxtgp\""
Sep 12 02:50:12.217555 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.217463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 02:50:12.220764 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.220744    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"]
Sep 12 02:50:12.239707 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.239682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.239812 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.239713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.239812 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.239731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hx4j\" (UniqueName: \"kubernetes.io/projected/5ae59272-69af-4bcc-89f8-9373e32bcda5-kube-api-access-8hx4j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.239925 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.239836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae59272-69af-4bcc-89f8-9373e32bcda5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.239925 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.239886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.240022 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.239983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.243699 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.243677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:12.340412 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae59272-69af-4bcc-89f8-9373e32bcda5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.340601 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.340601 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.340601 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.340601 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.340794 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hx4j\" (UniqueName: \"kubernetes.io/projected/5ae59272-69af-4bcc-89f8-9373e32bcda5-kube-api-access-8hx4j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.340965 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.341050 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.340999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.341206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.341188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.343228 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.343202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae59272-69af-4bcc-89f8-9373e32bcda5-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.343372 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.343354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae59272-69af-4bcc-89f8-9373e32bcda5-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.348833 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.348813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hx4j\" (UniqueName: \"kubernetes.io/projected/5ae59272-69af-4bcc-89f8-9373e32bcda5-kube-api-access-8hx4j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm\" (UID: \"5ae59272-69af-4bcc-89f8-9373e32bcda5\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.525420 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.525311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:12.797494 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.797400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"]
Sep 12 02:50:12.800479 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:50:12.800433    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae59272_69af_4bcc_89f8_9373e32bcda5.slice/crio-3cd3807cc11966990c92eacb7ff67b92b839ee5b9edbd041681d92e59abf5da0 WatchSource:0}: Error finding container 3cd3807cc11966990c92eacb7ff67b92b839ee5b9edbd041681d92e59abf5da0: Status 404 returned error can't find the container with id 3cd3807cc11966990c92eacb7ff67b92b839ee5b9edbd041681d92e59abf5da0
Sep 12 02:50:12.802081 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:12.802064    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:50:13.408531 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:13.408486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm" event={"ID":"5ae59272-69af-4bcc-89f8-9373e32bcda5","Type":"ContainerStarted","Data":"3cd3807cc11966990c92eacb7ff67b92b839ee5b9edbd041681d92e59abf5da0"}
Sep 12 02:50:14.419405 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:14.419366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm" event={"ID":"5ae59272-69af-4bcc-89f8-9373e32bcda5","Type":"ContainerStarted","Data":"4ad15e4a58554518496b7f52c7a8117036d09f5fbb5644d2c8a9e59ee56c8efb"}
Sep 12 02:50:14.436395 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:14.436345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm" podStartSLOduration=1.409562615 podStartE2EDuration="2.436330133s" totalImagesPullingTime="1.026767518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:50:14.435253622 +0000 UTC m=+832.718167204" watchObservedRunningTime="2026-09-12 02:50:14.436330133 +0000 UTC m=+832.719243714"
Sep 12 02:50:17.448511 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:17.448480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:22.526153 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:22.526121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:22.526618 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:22.526192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:22.538357 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:22.538335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:23.469525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:23.469501    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm"
Sep 12 02:50:29.152375 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:29.152341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:35.418841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.418805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"]
Sep 12 02:50:35.422914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.422895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.425602 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.425583    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 12 02:50:35.429841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.429815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"]
Sep 12 02:50:35.444860 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.444837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:35.546869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.546840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.547021 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.546880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a29b208-1fa2-4601-a6dd-3bf801054398-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.547021 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.546909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.547021 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.546986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.547138 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.547036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.547138 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.547058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntxq\" (UniqueName: \"kubernetes.io/projected/1a29b208-1fa2-4601-a6dd-3bf801054398-kube-api-access-5ntxq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648361 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648361 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntxq\" (UniqueName: \"kubernetes.io/projected/1a29b208-1fa2-4601-a6dd-3bf801054398-kube-api-access-5ntxq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a29b208-1fa2-4601-a6dd-3bf801054398-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648612 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648801 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648856 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.648856 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.648845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.650892 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.650871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a29b208-1fa2-4601-a6dd-3bf801054398-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.651165 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.651146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a29b208-1fa2-4601-a6dd-3bf801054398-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.657216 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.657196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntxq\" (UniqueName: \"kubernetes.io/projected/1a29b208-1fa2-4601-a6dd-3bf801054398-kube-api-access-5ntxq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cv2kn\" (UID: \"1a29b208-1fa2-4601-a6dd-3bf801054398\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:35.734233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:35.734149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:36.004372 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:36.004291    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"]
Sep 12 02:50:36.008139 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:50:36.008105    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a29b208_1fa2_4601_a6dd_3bf801054398.slice/crio-35493950c00e5b85ef7ae03e2233d0154e487aff28492275c27e5398e203509d WatchSource:0}: Error finding container 35493950c00e5b85ef7ae03e2233d0154e487aff28492275c27e5398e203509d: Status 404 returned error can't find the container with id 35493950c00e5b85ef7ae03e2233d0154e487aff28492275c27e5398e203509d
Sep 12 02:50:36.516526 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:36.516490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn" event={"ID":"1a29b208-1fa2-4601-a6dd-3bf801054398","Type":"ContainerStarted","Data":"595d50741284eb644a2f4cc701df3e95a96e567cd3dfb6ff4cc62269649d072c"}
Sep 12 02:50:36.516915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:36.516539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn" event={"ID":"1a29b208-1fa2-4601-a6dd-3bf801054398","Type":"ContainerStarted","Data":"35493950c00e5b85ef7ae03e2233d0154e487aff28492275c27e5398e203509d"}
Sep 12 02:50:36.533691 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:36.533645    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn" podStartSLOduration=1.347923094 podStartE2EDuration="1.533630532s" totalImagesPullingTime="185.707438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:50:36.532300564 +0000 UTC m=+854.815214147" watchObservedRunningTime="2026-09-12 02:50:36.533630532 +0000 UTC m=+854.816544114"
Sep 12 02:50:39.046867 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:39.046829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:39.096390 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:39.096362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:44.450131 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:44.450095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:45.734537 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:45.734504    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:45.734939 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:45.734650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:45.746926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:45.746901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:46.574645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:46.574617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cv2kn"
Sep 12 02:50:48.511805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:48.511773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:50:48.614677 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:50:48.614643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:51:22.177872 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:22.177790    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:51:22.178400 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:22.178309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:51:58.120171 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.120137    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d5fbf76c7-fqmkz"]
Sep 12 02:51:58.123611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.123589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.130309 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.130278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d5fbf76c7-fqmkz"]
Sep 12 02:51:58.139527 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.139498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvxv\" (UniqueName: \"kubernetes.io/projected/eceb89ea-7b8d-4f58-82c7-e55025fa4650-kube-api-access-bkvxv\") pod \"authorino-7d5fbf76c7-fqmkz\" (UID: \"eceb89ea-7b8d-4f58-82c7-e55025fa4650\") " pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.139643 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.139545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eceb89ea-7b8d-4f58-82c7-e55025fa4650-tls-cert\") pod \"authorino-7d5fbf76c7-fqmkz\" (UID: \"eceb89ea-7b8d-4f58-82c7-e55025fa4650\") " pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.240190 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.240150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eceb89ea-7b8d-4f58-82c7-e55025fa4650-tls-cert\") pod \"authorino-7d5fbf76c7-fqmkz\" (UID: \"eceb89ea-7b8d-4f58-82c7-e55025fa4650\") " pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.240362 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.240288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvxv\" (UniqueName: \"kubernetes.io/projected/eceb89ea-7b8d-4f58-82c7-e55025fa4650-kube-api-access-bkvxv\") pod \"authorino-7d5fbf76c7-fqmkz\" (UID: \"eceb89ea-7b8d-4f58-82c7-e55025fa4650\") " pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.242833 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.242805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eceb89ea-7b8d-4f58-82c7-e55025fa4650-tls-cert\") pod \"authorino-7d5fbf76c7-fqmkz\" (UID: \"eceb89ea-7b8d-4f58-82c7-e55025fa4650\") " pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.248320 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.248299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvxv\" (UniqueName: \"kubernetes.io/projected/eceb89ea-7b8d-4f58-82c7-e55025fa4650-kube-api-access-bkvxv\") pod \"authorino-7d5fbf76c7-fqmkz\" (UID: \"eceb89ea-7b8d-4f58-82c7-e55025fa4650\") " pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.434364 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.434287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz"
Sep 12 02:51:58.702757 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.702735    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d5fbf76c7-fqmkz"]
Sep 12 02:51:58.705815 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:51:58.705789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeceb89ea_7b8d_4f58_82c7_e55025fa4650.slice/crio-f7b923f637fe3ecd2557eb2734515fde239839d6c035c196720320d89ae6f814 WatchSource:0}: Error finding container f7b923f637fe3ecd2557eb2734515fde239839d6c035c196720320d89ae6f814: Status 404 returned error can't find the container with id f7b923f637fe3ecd2557eb2734515fde239839d6c035c196720320d89ae6f814
Sep 12 02:51:58.894397 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:58.894369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz" event={"ID":"eceb89ea-7b8d-4f58-82c7-e55025fa4650","Type":"ContainerStarted","Data":"f7b923f637fe3ecd2557eb2734515fde239839d6c035c196720320d89ae6f814"}
Sep 12 02:51:59.902521 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:59.902439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz" event={"ID":"eceb89ea-7b8d-4f58-82c7-e55025fa4650","Type":"ContainerStarted","Data":"a871375eb60918c895a2c227f0bb348079b511f80db88738525ea67d9cc38dc5"}
Sep 12 02:51:59.918223 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:59.918169    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d5fbf76c7-fqmkz" podStartSLOduration=1.164642556 podStartE2EDuration="1.918153823s" totalImagesPullingTime="753.511267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:51:59.916893846 +0000 UTC m=+938.199807428" watchObservedRunningTime="2026-09-12 02:51:59.918153823 +0000 UTC m=+938.201067403"
Sep 12 02:51:59.947137 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:59.947103    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b67b44488-s85qd"]
Sep 12 02:51:59.947577 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:51:59.947550    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b67b44488-s85qd" podUID="39c6a227-9d1a-4f3d-9018-35b9862ecb32" containerName="authorino" containerID="cri-o://baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e" gracePeriod=30
Sep 12 02:52:00.199895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.199873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:52:00.257328 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.256776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vpth\" (UniqueName: \"kubernetes.io/projected/39c6a227-9d1a-4f3d-9018-35b9862ecb32-kube-api-access-5vpth\") pod \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") "
Sep 12 02:52:00.257328 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.256826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c6a227-9d1a-4f3d-9018-35b9862ecb32-tls-cert\") pod \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\" (UID: \"39c6a227-9d1a-4f3d-9018-35b9862ecb32\") "
Sep 12 02:52:00.259909 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.259875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c6a227-9d1a-4f3d-9018-35b9862ecb32-kube-api-access-5vpth" (OuterVolumeSpecName: "kube-api-access-5vpth") pod "39c6a227-9d1a-4f3d-9018-35b9862ecb32" (UID: "39c6a227-9d1a-4f3d-9018-35b9862ecb32"). InnerVolumeSpecName "kube-api-access-5vpth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:52:00.267938 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.267910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c6a227-9d1a-4f3d-9018-35b9862ecb32-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "39c6a227-9d1a-4f3d-9018-35b9862ecb32" (UID: "39c6a227-9d1a-4f3d-9018-35b9862ecb32"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:52:00.357725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.357683    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpth\" (UniqueName: \"kubernetes.io/projected/39c6a227-9d1a-4f3d-9018-35b9862ecb32-kube-api-access-5vpth\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:52:00.357725 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.357712    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c6a227-9d1a-4f3d-9018-35b9862ecb32-tls-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:52:00.559551 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.559486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:00.654679 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.654637    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:00.908295 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.908261    2579 generic.go:356] "Generic (PLEG): container finished" podID="39c6a227-9d1a-4f3d-9018-35b9862ecb32" containerID="baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e" exitCode=0
Sep 12 02:52:00.908766 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.908305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b67b44488-s85qd"
Sep 12 02:52:00.908766 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.908345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b67b44488-s85qd" event={"ID":"39c6a227-9d1a-4f3d-9018-35b9862ecb32","Type":"ContainerDied","Data":"baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e"}
Sep 12 02:52:00.908766 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.908390    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b67b44488-s85qd" event={"ID":"39c6a227-9d1a-4f3d-9018-35b9862ecb32","Type":"ContainerDied","Data":"d5024cd970ee818653e49cc2e86f1df97db5d4f56af876d4c712977166e3503a"}
Sep 12 02:52:00.908766 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.908414    2579 scope.go:117] "RemoveContainer" containerID="baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e"
Sep 12 02:52:00.918540 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.918523    2579 scope.go:117] "RemoveContainer" containerID="baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e"
Sep 12 02:52:00.919231 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:52:00.919213    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e\": container with ID starting with baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e not found: ID does not exist" containerID="baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e"
Sep 12 02:52:00.919311 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.919243    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e"} err="failed to get container status \"baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e\": rpc error: code = NotFound desc = could not find container \"baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e\": container with ID starting with baa431fa32bc1777d4e0c60f4f34adbbc2b5ee338d79db8c962c788d7644e45e not found: ID does not exist"
Sep 12 02:52:00.929660 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.929639    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b67b44488-s85qd"]
Sep 12 02:52:00.935841 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:00.935814    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b67b44488-s85qd"]
Sep 12 02:52:01.977611 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:01.977578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:02.078696 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:02.078656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:02.245531 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:02.245433    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c6a227-9d1a-4f3d-9018-35b9862ecb32" path="/var/lib/kubelet/pods/39c6a227-9d1a-4f3d-9018-35b9862ecb32/volumes"
Sep 12 02:52:18.809614 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:18.809584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:18.907616 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:18.907574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.011228 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.011184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.170029 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.169997    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.204643 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.204612    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"]
Sep 12 02:52:19.204961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.204948    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c6a227-9d1a-4f3d-9018-35b9862ecb32" containerName="authorino"
Sep 12 02:52:19.205039 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.204962    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c6a227-9d1a-4f3d-9018-35b9862ecb32" containerName="authorino"
Sep 12 02:52:19.205039 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.205023    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c6a227-9d1a-4f3d-9018-35b9862ecb32" containerName="authorino"
Sep 12 02:52:19.207892 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.207871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.210557 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.210535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b0573c-openshift-default-dockercfg-h982j\""
Sep 12 02:52:19.222116 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.222092    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"]
Sep 12 02:52:19.248274 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.248240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.316832 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.316792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trls4\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-kube-api-access-trls4\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.316986 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.316844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-credential-socket\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.316986 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.316877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-token\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.316986 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.316921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574e6656-2c15-44ef-8f77-27a10fc503b3-istiod-ca-cert\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.316986 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.316958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-socket\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.317125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.317041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-envoy\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.317125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.317069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-certs\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.317125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.317102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-podinfo\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.317219 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.317138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-data\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.325416 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.325393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.385805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.385773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.417795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trls4\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-kube-api-access-trls4\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.417961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-credential-socket\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.417961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-token\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.417961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574e6656-2c15-44ef-8f77-27a10fc503b3-istiod-ca-cert\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.417961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-socket\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.417961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-envoy\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418254 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-certs\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418254 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.417990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-podinfo\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418254 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.418023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-data\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418254 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.418242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-credential-socket\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418485 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.418357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-data\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418485 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.418435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-socket\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.418526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-certs\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.418857 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.418833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574e6656-2c15-44ef-8f77-27a10fc503b3-istiod-ca-cert\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.420493 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.420422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-envoy\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.420751 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.420734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-podinfo\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.425874 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.425854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trls4\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-kube-api-access-trls4\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.426044 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.426027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-token\") pod \"e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") " pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.487009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.486973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:19.518846 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.518813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:19.791384 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.791356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"]
Sep 12 02:52:19.793772 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:19.793745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574e6656_2c15_44ef_8f77_27a10fc503b3.slice/crio-497f39b914eef2cd4f9b64d340082c62fbb63415f5e1bc1d31eaa4910dae6a88 WatchSource:0}: Error finding container 497f39b914eef2cd4f9b64d340082c62fbb63415f5e1bc1d31eaa4910dae6a88: Status 404 returned error can't find the container with id 497f39b914eef2cd4f9b64d340082c62fbb63415f5e1bc1d31eaa4910dae6a88
Sep 12 02:52:19.796318 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.796288    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:19.796384 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.796359    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:19.796421 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.796387    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:19.999744 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.999701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" event={"ID":"574e6656-2c15-44ef-8f77-27a10fc503b3","Type":"ContainerStarted","Data":"9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"}
Sep 12 02:52:20.000125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:19.999762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" event={"ID":"574e6656-2c15-44ef-8f77-27a10fc503b3","Type":"ContainerStarted","Data":"497f39b914eef2cd4f9b64d340082c62fbb63415f5e1bc1d31eaa4910dae6a88"}
Sep 12 02:52:20.018821 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:20.018779    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podStartSLOduration=1.018763831 podStartE2EDuration="1.018763831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:20.017547922 +0000 UTC m=+958.300461495" watchObservedRunningTime="2026-09-12 02:52:20.018763831 +0000 UTC m=+958.301677411"
Sep 12 02:52:20.481804 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:20.481737    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:20.519292 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:20.519261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:20.523878 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:20.523855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:20.566607 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:20.566576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:21.008201 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:21.008173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:21.009046 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:21.009022    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:21.016193 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:21.016169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:22.132590 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:22.132553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:22.227750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:22.227708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:22.653700 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:22.653667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:22.759918 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:22.759879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:22.883697 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:22.883661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:22.978205 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:22.978130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:23.643664 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.643627    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"]
Sep 12 02:52:23.648806 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.648783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.658912 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.657825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b0573c-serving-cert\""
Sep 12 02:52:23.666389 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.664365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"]
Sep 12 02:52:23.730378 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.730341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:23.759251 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.759214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-metrics-certs\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.759251 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.759251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067a47dc-2c59-4898-bf89-5b8063f5f90a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.759558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.759272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.759558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.759339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9sch\" (UniqueName: \"kubernetes.io/projected/067a47dc-2c59-4898-bf89-5b8063f5f90a-kube-api-access-s9sch\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.830101 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.830061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:23.859837 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.859804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9sch\" (UniqueName: \"kubernetes.io/projected/067a47dc-2c59-4898-bf89-5b8063f5f90a-kube-api-access-s9sch\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.860074 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.859905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-metrics-certs\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.860074 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.859942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067a47dc-2c59-4898-bf89-5b8063f5f90a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.860074 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.859969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.862776 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.860798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067a47dc-2c59-4898-bf89-5b8063f5f90a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.866498 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.866440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-metrics-certs\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.866498 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.866484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.868470 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.868418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9sch\" (UniqueName: \"kubernetes.io/projected/067a47dc-2c59-4898-bf89-5b8063f5f90a-kube-api-access-s9sch\") pod \"maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:23.930666 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.930572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:23.971848 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:23.971807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:24.117308 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.117270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:24.248576 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.248553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"]
Sep 12 02:52:24.250720 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:24.250690    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067a47dc_2c59_4898_bf89_5b8063f5f90a.slice/crio-0b4eeb818a207042cd8f8d97c20faac59ccd829a36976671e24cfff4a8f863ef WatchSource:0}: Error finding container 0b4eeb818a207042cd8f8d97c20faac59ccd829a36976671e24cfff4a8f863ef: Status 404 returned error can't find the container with id 0b4eeb818a207042cd8f8d97c20faac59ccd829a36976671e24cfff4a8f863ef
Sep 12 02:52:24.440009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.439969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:24.543058 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.542950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:24.579884 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.579857    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"]
Sep 12 02:52:24.584640 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.584622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.587400 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.587382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b0573c-dockercfg-86nrg\""
Sep 12 02:52:24.587519 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.587415    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b0573c\""
Sep 12 02:52:24.593532 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.593510    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"]
Sep 12 02:52:24.669086 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.669053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkfw\" (UniqueName: \"kubernetes.io/projected/9de4c729-3351-428c-8cc3-78ed7cd2c751-kube-api-access-7lkfw\") pod \"payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.669518 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.669093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de4c729-3351-428c-8cc3-78ed7cd2c751-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.770009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.769976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkfw\" (UniqueName: \"kubernetes.io/projected/9de4c729-3351-428c-8cc3-78ed7cd2c751-kube-api-access-7lkfw\") pod \"payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.770009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.770021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de4c729-3351-428c-8cc3-78ed7cd2c751-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.770792 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.770773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de4c729-3351-428c-8cc3-78ed7cd2c751-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.778406 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.778384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkfw\" (UniqueName: \"kubernetes.io/projected/9de4c729-3351-428c-8cc3-78ed7cd2c751-kube-api-access-7lkfw\") pod \"payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:24.846823 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.846716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:24.896823 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:24.896775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:25.032568 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.032535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62" event={"ID":"067a47dc-2c59-4898-bf89-5b8063f5f90a","Type":"ContainerStarted","Data":"d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83"}
Sep 12 02:52:25.032754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.032589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62" event={"ID":"067a47dc-2c59-4898-bf89-5b8063f5f90a","Type":"ContainerStarted","Data":"0b4eeb818a207042cd8f8d97c20faac59ccd829a36976671e24cfff4a8f863ef"}
Sep 12 02:52:25.032754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.032651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:25.051223 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.051175    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62" podStartSLOduration=1.673317685 podStartE2EDuration="2.051159519s" totalImagesPullingTime="377.841834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:25.049527012 +0000 UTC m=+963.332440593" watchObservedRunningTime="2026-09-12 02:52:25.051159519 +0000 UTC m=+963.334073100"
Sep 12 02:52:25.197660 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.197593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"]
Sep 12 02:52:25.201999 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:25.201968    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de4c729_3351_428c_8cc3_78ed7cd2c751.slice/crio-deb342baff9c68db63f19ca8761dfc028c67c61a6d6fdb44dafacc174a86f25a WatchSource:0}: Error finding container deb342baff9c68db63f19ca8761dfc028c67c61a6d6fdb44dafacc174a86f25a: Status 404 returned error can't find the container with id deb342baff9c68db63f19ca8761dfc028c67c61a6d6fdb44dafacc174a86f25a
Sep 12 02:52:25.756119 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.756090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:25.811549 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.811516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:25.928707 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:25.928674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:26.040416 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.040332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6" event={"ID":"9de4c729-3351-428c-8cc3-78ed7cd2c751","Type":"ContainerStarted","Data":"1ca659c7223c850d6f404f8810b4ed7938728f96ba3c60bf55855045d43ef84e"}
Sep 12 02:52:26.040416 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.040384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6" event={"ID":"9de4c729-3351-428c-8cc3-78ed7cd2c751","Type":"ContainerStarted","Data":"deb342baff9c68db63f19ca8761dfc028c67c61a6d6fdb44dafacc174a86f25a"}
Sep 12 02:52:26.040658 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.040641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:26.059816 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.059772    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6" podStartSLOduration=2.059757091 podStartE2EDuration="2.059757091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:26.05841121 +0000 UTC m=+964.341324792" watchObservedRunningTime="2026-09-12 02:52:26.059757091 +0000 UTC m=+964.342670672"
Sep 12 02:52:26.206970 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.206941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:26.313332 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.313252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:26.359855 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.359827    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"]
Sep 12 02:52:26.363486 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.363460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.366195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.366174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c6190b-dockercfg-9mfhd\""
Sep 12 02:52:26.366195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.366184    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c6190b\""
Sep 12 02:52:26.372033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.372013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"]
Sep 12 02:52:26.486398 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.486357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7fd623-5c80-42df-a7c5-ad74730d720d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.486398 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.486401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchq2\" (UniqueName: \"kubernetes.io/projected/1e7fd623-5c80-42df-a7c5-ad74730d720d-kube-api-access-kchq2\") pod \"payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.587204 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.587125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7fd623-5c80-42df-a7c5-ad74730d720d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.587204 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.587164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kchq2\" (UniqueName: \"kubernetes.io/projected/1e7fd623-5c80-42df-a7c5-ad74730d720d-kube-api-access-kchq2\") pod \"payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.587825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.587807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7fd623-5c80-42df-a7c5-ad74730d720d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.598075 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.598044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchq2\" (UniqueName: \"kubernetes.io/projected/1e7fd623-5c80-42df-a7c5-ad74730d720d-kube-api-access-kchq2\") pod \"payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.608474 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.608432    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"]
Sep 12 02:52:26.612117 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.612094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.614772 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.614751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a07b10-serving-cert\""
Sep 12 02:52:26.627919 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.627895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"]
Sep 12 02:52:26.645831 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.645799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:26.675035 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.675002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:26.688467 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.688416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b378e1a-dc2b-4357-8ebe-e901c475411d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.688597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.688480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmxb\" (UniqueName: \"kubernetes.io/projected/2b378e1a-dc2b-4357-8ebe-e901c475411d-kube-api-access-9pmxb\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.688597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.688515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-metrics-certs\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.688852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.688601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-maas-api-tls\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.701216 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.701187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:26.789268 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.789236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-maas-api-tls\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.789693 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.789335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b378e1a-dc2b-4357-8ebe-e901c475411d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.789693 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.789357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmxb\" (UniqueName: \"kubernetes.io/projected/2b378e1a-dc2b-4357-8ebe-e901c475411d-kube-api-access-9pmxb\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.789693 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.789375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-metrics-certs\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.790073 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.790052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b378e1a-dc2b-4357-8ebe-e901c475411d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.791894 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.791877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-metrics-certs\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.792059 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.792042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-maas-api-tls\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.798615 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.798589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmxb\" (UniqueName: \"kubernetes.io/projected/2b378e1a-dc2b-4357-8ebe-e901c475411d-kube-api-access-9pmxb\") pod \"maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.802026 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.801766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:26.922334 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.922298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:26.951147 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:26.951126    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"]
Sep 12 02:52:26.952876 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:26.952840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7fd623_5c80_42df_a7c5_ad74730d720d.slice/crio-291b70fca3e3e1da15367f614ee583d3b7d2e05873006f76b0b0bc3b24859277 WatchSource:0}: Error finding container 291b70fca3e3e1da15367f614ee583d3b7d2e05873006f76b0b0bc3b24859277: Status 404 returned error can't find the container with id 291b70fca3e3e1da15367f614ee583d3b7d2e05873006f76b0b0bc3b24859277
Sep 12 02:52:27.047770 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.047733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w" event={"ID":"1e7fd623-5c80-42df-a7c5-ad74730d720d","Type":"ContainerStarted","Data":"330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56"}
Sep 12 02:52:27.047989 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.047798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w" event={"ID":"1e7fd623-5c80-42df-a7c5-ad74730d720d","Type":"ContainerStarted","Data":"291b70fca3e3e1da15367f614ee583d3b7d2e05873006f76b0b0bc3b24859277"}
Sep 12 02:52:27.048073 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.048055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:27.066405 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.066331    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w" podStartSLOduration=1.066308857 podStartE2EDuration="1.066308857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:27.064993887 +0000 UTC m=+965.347907472" watchObservedRunningTime="2026-09-12 02:52:27.066308857 +0000 UTC m=+965.349222439"
Sep 12 02:52:27.143558 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.143526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:27.203949 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.203928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"]
Sep 12 02:52:27.206844 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:27.206804    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b378e1a_dc2b_4357_8ebe_e901c475411d.slice/crio-7118fbf16cd65c4cbfe19e3756df58bd97a7b71ed8036df6b8b81dfb983c37b9 WatchSource:0}: Error finding container 7118fbf16cd65c4cbfe19e3756df58bd97a7b71ed8036df6b8b81dfb983c37b9: Status 404 returned error can't find the container with id 7118fbf16cd65c4cbfe19e3756df58bd97a7b71ed8036df6b8b81dfb983c37b9
Sep 12 02:52:27.207759 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.207729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:27.683525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.683489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:27.787559 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:27.787480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:28.008627 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.008588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"]
Sep 12 02:52:28.012639 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.012610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.015478 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.015432    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4b3be7-openshift-default-dockercfg-ctkgf\""
Sep 12 02:52:28.024501 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.024477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"]
Sep 12 02:52:28.047372 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.047303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:28.054808 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.054779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj" event={"ID":"2b378e1a-dc2b-4357-8ebe-e901c475411d","Type":"ContainerStarted","Data":"7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595"}
Sep 12 02:52:28.054941 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.054829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj" event={"ID":"2b378e1a-dc2b-4357-8ebe-e901c475411d","Type":"ContainerStarted","Data":"7118fbf16cd65c4cbfe19e3756df58bd97a7b71ed8036df6b8b81dfb983c37b9"}
Sep 12 02:52:28.075329 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.075282    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj" podStartSLOduration=1.7997135869999998 podStartE2EDuration="2.075262508s" totalImagesPullingTime="275.548921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:28.074752764 +0000 UTC m=+966.357666372" watchObservedRunningTime="2026-09-12 02:52:28.075262508 +0000 UTC m=+966.358176090"
Sep 12 02:52:28.103355 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-data\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istiod-ca-cert\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-podinfo\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-socket\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqll\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-kube-api-access-cvqll\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-envoy\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-credential-socket\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-certs\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.103717 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.103678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-token\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.204924 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.204875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-data\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205111 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.204948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istiod-ca-cert\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205111 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-podinfo\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205111 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-socket\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqll\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-kube-api-access-cvqll\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-envoy\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-credential-socket\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-certs\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-token\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-data\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.205838 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.205764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istiod-ca-cert\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.206265 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.206110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-socket\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.206265 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.206242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-credential-socket\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.206538 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.206511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-certs\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.209031 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.209008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-envoy\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.209031 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.209020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-podinfo\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.214200 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.214169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqll\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-kube-api-access-cvqll\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.214379 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.214357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-token\") pod \"e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") " pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.324638 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.324562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:28.331363 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.331339    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:28.449319 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.449288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:28.535635 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.535598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:28.609632 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.609606    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"]
Sep 12 02:52:28.611178 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:28.611146    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0f90e8_ca5c_40e4_99be_ac759aa73777.slice/crio-be93889bfd80d834e1d28c05a79e23cc6be9db3d6ae59eb27829d7bd50a32053 WatchSource:0}: Error finding container be93889bfd80d834e1d28c05a79e23cc6be9db3d6ae59eb27829d7bd50a32053: Status 404 returned error can't find the container with id be93889bfd80d834e1d28c05a79e23cc6be9db3d6ae59eb27829d7bd50a32053
Sep 12 02:52:28.611294 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.611253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:28.613570 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.613544    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:28.613641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.613605    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:28.613641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.613634    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:28.891976 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:28.891889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:29.061685 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.061647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" event={"ID":"9d0f90e8-ca5c-40e4-99be-ac759aa73777","Type":"ContainerStarted","Data":"63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"}
Sep 12 02:52:29.062152 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.061713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" event={"ID":"9d0f90e8-ca5c-40e4-99be-ac759aa73777","Type":"ContainerStarted","Data":"be93889bfd80d834e1d28c05a79e23cc6be9db3d6ae59eb27829d7bd50a32053"}
Sep 12 02:52:29.062152 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.061748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:29.101658 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.101612    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podStartSLOduration=2.101595755 podStartE2EDuration="2.101595755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:29.099755921 +0000 UTC m=+967.382669501" watchObservedRunningTime="2026-09-12 02:52:29.101595755 +0000 UTC m=+967.384509336"
Sep 12 02:52:29.324894 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.324811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:29.329753 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.329717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:29.339196 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.339171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:29.456836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:29.456801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:30.066535 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:30.066501    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:30.067503 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:30.067468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:30.309512 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:30.309412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:31.017622 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:31.017566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:31.046337 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:31.046305    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 02:52:31.115944 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:31.115907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:31.223805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:31.223757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:31.310325 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:31.310223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:32.298659 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:32.298626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:33.291594 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:33.291561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:34.171506 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.171402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:34.229417 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.229373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:34.332072 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.332041    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:34.659759 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.659733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:34.737728 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.737678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:34.787280 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.787243    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"]
Sep 12 02:52:34.801308 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.801264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:34.802891 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.802869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"]
Sep 12 02:52:34.804137 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.804119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-214c0e42-dockercfg-bwshz\""
Sep 12 02:52:34.804243 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.804211    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-214c0e42\""
Sep 12 02:52:34.982754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.982647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949eb71-540c-450c-92aa-3ebea5df46af-plugins-config-volume\") pod \"payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") " pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:34.982754 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:34.982721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5l8\" (UniqueName: \"kubernetes.io/projected/b949eb71-540c-450c-92aa-3ebea5df46af-kube-api-access-rl5l8\") pod \"payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") " pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:35.083785 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.083748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949eb71-540c-450c-92aa-3ebea5df46af-plugins-config-volume\") pod \"payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") " pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:35.084079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.084058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5l8\" (UniqueName: \"kubernetes.io/projected/b949eb71-540c-450c-92aa-3ebea5df46af-kube-api-access-rl5l8\") pod \"payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") " pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:35.085413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.085388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949eb71-540c-450c-92aa-3ebea5df46af-plugins-config-volume\") pod \"payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") " pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:35.086017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.085992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:35.095015 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.094985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:52:35.099396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.099362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5l8\" (UniqueName: \"kubernetes.io/projected/b949eb71-540c-450c-92aa-3ebea5df46af-kube-api-access-rl5l8\") pod \"payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") " pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:35.113365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.113304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:35.145808 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.145674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:35.418598 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.418574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"]
Sep 12 02:52:35.420325 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:35.420297    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb949eb71_540c_450c_92aa_3ebea5df46af.slice/crio-7e0bcc903595b58d8e28f01bb06d7376a43aaa3373525c91b34fe9069d76f526 WatchSource:0}: Error finding container 7e0bcc903595b58d8e28f01bb06d7376a43aaa3373525c91b34fe9069d76f526: Status 404 returned error can't find the container with id 7e0bcc903595b58d8e28f01bb06d7376a43aaa3373525c91b34fe9069d76f526
Sep 12 02:52:35.501955 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.501926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:35.681059 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.680907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:35.771471 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:35.771065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:36.108583 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:36.108553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64" event={"ID":"b949eb71-540c-450c-92aa-3ebea5df46af","Type":"ContainerStarted","Data":"711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2"}
Sep 12 02:52:36.108734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:36.108611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64" event={"ID":"b949eb71-540c-450c-92aa-3ebea5df46af","Type":"ContainerStarted","Data":"7e0bcc903595b58d8e28f01bb06d7376a43aaa3373525c91b34fe9069d76f526"}
Sep 12 02:52:36.108734 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:36.108658    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:36.125350 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:36.125303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64" podStartSLOduration=2.125288414 podStartE2EDuration="2.125288414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:36.123500447 +0000 UTC m=+974.406414030" watchObservedRunningTime="2026-09-12 02:52:36.125288414 +0000 UTC m=+974.408202008"
Sep 12 02:52:36.175827 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:36.175788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:36.835244 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:36.835203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:37.049884 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:37.049855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 02:52:37.095404 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:37.095293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:37.298239 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:37.298202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:37.398249 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:37.398216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:37.500986 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:37.500940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:37.964811 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:37.964782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:38.057668 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:38.057595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 02:52:38.072934 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:38.072890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:38.163175 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:38.163141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:38.263696 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:38.263660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:38.363799 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:38.363769    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:38.926032 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:38.926002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:39.767023 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:39.766982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:40.637671 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:40.637627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:41.066242 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:41.066099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:41.397974 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:41.397939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:42.153322 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.153283    2579 generic.go:356] "Generic (PLEG): container finished" podID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerID="9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640" exitCode=0
Sep 12 02:52:42.153761 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.153356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" event={"ID":"574e6656-2c15-44ef-8f77-27a10fc503b3","Type":"ContainerDied","Data":"9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"}
Sep 12 02:52:42.153761 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.153720    2579 scope.go:117] "RemoveContainer" containerID="9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"
Sep 12 02:52:42.154832 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.154808    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:42.154890 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.154882    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:42.154944 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.154932    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:42.478659 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.478559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:42.583767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.583735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:42.678876 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.678824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:42.780261 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.780145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:42.940953 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:42.940903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:43.053954 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.053859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:43.151744 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.151711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:43.161074 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.161041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" event={"ID":"574e6656-2c15-44ef-8f77-27a10fc503b3","Type":"ContainerStarted","Data":"d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361"}
Sep 12 02:52:43.270412 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.270380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:43.353506 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.353413    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"]
Sep 12 02:52:43.357007 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.356990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.360235 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.360214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f93121-openshift-default-dockercfg-hvk6c\""
Sep 12 02:52:43.389946 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.389915    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"]
Sep 12 02:52:43.421498 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.419021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:43.487420 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.487382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-data\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.487735 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.487708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-podinfo\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.487873 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.487859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-socket\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.487981 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.487968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istiod-ca-cert\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.488122 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.488108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-envoy\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.488230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.488216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-credential-socket\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.488348 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.488333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-token\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.488480 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.488442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-certs\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.488596 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.488583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86n5j\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-kube-api-access-86n5j\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.521476 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.519704    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:43.521476 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.519935    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 12 02:52:43.521476 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.520005    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 12 02:52:43.590176 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-socket\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istiod-ca-cert\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-envoy\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-credential-socket\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-token\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-certs\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590393 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86n5j\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-kube-api-access-86n5j\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590762 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-data\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.590762 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.590517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-podinfo\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.591413 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.591387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-credential-socket\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.592732 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.592673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istiod-ca-cert\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.593187 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.593071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-socket\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.593354 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.593336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-certs\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.594112 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.594067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-data\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.595710 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.594929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-podinfo\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.597961 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.597939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-envoy\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.603556 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.603426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86n5j\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-kube-api-access-86n5j\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.604125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.604104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-token\") pod \"e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") " pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.670990 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.670894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:43.698321 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.698222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:43.937898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:43.937861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:44.052003 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.051979    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"]
Sep 12 02:52:44.054131 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:52:44.054107    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57f3ab4_773a_4f5b_a706_bb0e272868b3.slice/crio-fc320679c64a0ac59bac32c640a67ea61de499129f0c03c086ccce61aa877a96 WatchSource:0}: Error finding container fc320679c64a0ac59bac32c640a67ea61de499129f0c03c086ccce61aa877a96: Status 404 returned error can't find the container with id fc320679c64a0ac59bac32c640a67ea61de499129f0c03c086ccce61aa877a96
Sep 12 02:52:44.057161 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.057122    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:44.057235 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.057209    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:44.057273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.057255    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:44.198287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.198197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" event={"ID":"b57f3ab4-773a-4f5b-a706-bb0e272868b3","Type":"ContainerStarted","Data":"077dceae8828c79bdaaf46d5536e900aa0fdf313f4a940d81754c4dff448db6d"}
Sep 12 02:52:44.198767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.198301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" event={"ID":"b57f3ab4-773a-4f5b-a706-bb0e272868b3","Type":"ContainerStarted","Data":"fc320679c64a0ac59bac32c640a67ea61de499129f0c03c086ccce61aa877a96"}
Sep 12 02:52:44.224387 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.224322    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podStartSLOduration=1.224299608 podStartE2EDuration="1.224299608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:44.221701009 +0000 UTC m=+982.504614591" watchObservedRunningTime="2026-09-12 02:52:44.224299608 +0000 UTC m=+982.507213192"
Sep 12 02:52:44.519410 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.519300    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 12 02:52:44.519410 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.519375    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 12 02:52:44.699238 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.699190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:44.805834 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.805679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:44.907688 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:44.907564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:45.006995 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:45.006872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:45.104349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:45.104208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:45.519936 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:45.519897    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 12 02:52:45.520350 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:45.519952    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 12 02:52:45.713081 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:45.713050    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:46.181730 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.181700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:46.212973 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.212947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:46.215057 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.215038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:52:46.308810 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.308768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:46.414736 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.414706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:46.520248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.520151    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 12 02:52:46.520248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:46.520226    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 12 02:52:47.118046 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:47.118016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:52:47.149756 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:47.149726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:47.519909 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:47.519820    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 12 02:52:47.519909 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:47.519892    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 12 02:52:48.090939 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:48.090894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:48.610846 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:48.610781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:48.611657 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:48.611628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:48.613633 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:48.613611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 02:52:49.048983 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:49.048880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:49.745148 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:49.745114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:50.241405 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.241327    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerID="63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781" exitCode=0
Sep 12 02:52:50.249577 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.249550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" event={"ID":"9d0f90e8-ca5c-40e4-99be-ac759aa73777","Type":"ContainerDied","Data":"63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"}
Sep 12 02:52:50.249966 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.249949    2579 scope.go:117] "RemoveContainer" containerID="63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"
Sep 12 02:52:50.251079 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.251050    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:50.251177 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.251117    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:50.251177 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.251150    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:52:50.575284 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.575206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:50.699343 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.699305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:50.788308 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.788266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:50.921844 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:50.921812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:51.120555 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.120526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:51.218995 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.218913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:51.248521 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.248489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" event={"ID":"9d0f90e8-ca5c-40e4-99be-ac759aa73777","Type":"ContainerStarted","Data":"3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76"}
Sep 12 02:52:51.269125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.269083    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"]
Sep 12 02:52:51.273788 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.273755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.278906 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.278868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f93121-dockercfg-nhvrd\""
Sep 12 02:52:51.279066 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.278873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f93121\""
Sep 12 02:52:51.287966 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.287928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"]
Sep 12 02:52:51.300857 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.300824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"]
Sep 12 02:52:51.304240 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.304218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.317770 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.317749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"]
Sep 12 02:52:51.325750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.325724    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:51.326106 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.326064    2579 patch_prober.go:28] interesting pod/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 12 02:52:51.326250 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.326157    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 12 02:52:51.397073 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.397035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dc4e4e-28c8-486d-b99d-d765e7f28a98-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.397439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.397412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rln\" (UniqueName: \"kubernetes.io/projected/41dc4e4e-28c8-486d-b99d-d765e7f28a98-kube-api-access-n7rln\") pod \"payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.397565 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.397475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e414f93-adf3-41a5-b49f-a4ec1befe24e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.397629 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.397561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpcs\" (UniqueName: \"kubernetes.io/projected/9e414f93-adf3-41a5-b49f-a4ec1befe24e-kube-api-access-jhpcs\") pod \"payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.498351 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.498260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dc4e4e-28c8-486d-b99d-d765e7f28a98-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.498351 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.498296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rln\" (UniqueName: \"kubernetes.io/projected/41dc4e4e-28c8-486d-b99d-d765e7f28a98-kube-api-access-n7rln\") pod \"payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.498351 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.498313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e414f93-adf3-41a5-b49f-a4ec1befe24e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.498351 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.498346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpcs\" (UniqueName: \"kubernetes.io/projected/9e414f93-adf3-41a5-b49f-a4ec1befe24e-kube-api-access-jhpcs\") pod \"payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.499369 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.499346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dc4e4e-28c8-486d-b99d-d765e7f28a98-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.499519 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.499412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e414f93-adf3-41a5-b49f-a4ec1befe24e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.507219 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.506760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rln\" (UniqueName: \"kubernetes.io/projected/41dc4e4e-28c8-486d-b99d-d765e7f28a98-kube-api-access-n7rln\") pod \"payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.507330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.507228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpcs\" (UniqueName: \"kubernetes.io/projected/9e414f93-adf3-41a5-b49f-a4ec1befe24e-kube-api-access-jhpcs\") pod \"payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.591933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.591676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:51.617089 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.617062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:51.649946 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.649917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:51.744001 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.743976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:51.824300 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.824220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:51.870296 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.870240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"]
Sep 12 02:52:51.930200 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:51.930179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"]
Sep 12 02:52:52.007160 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.007132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:52.284830 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.284786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf" event={"ID":"41dc4e4e-28c8-486d-b99d-d765e7f28a98","Type":"ContainerStarted","Data":"cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547"}
Sep 12 02:52:52.285161 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.285137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf" event={"ID":"41dc4e4e-28c8-486d-b99d-d765e7f28a98","Type":"ContainerStarted","Data":"ff20ec00fe349a92fbfbbe874623a027743e7f26c2b2054ace25e630874dc9f2"}
Sep 12 02:52:52.285590 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.285537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:52:52.294468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.293437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh" event={"ID":"9e414f93-adf3-41a5-b49f-a4ec1befe24e","Type":"ContainerStarted","Data":"322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f"}
Sep 12 02:52:52.294468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.293500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh" event={"ID":"9e414f93-adf3-41a5-b49f-a4ec1befe24e","Type":"ContainerStarted","Data":"ac2772fb4e14013e09becf0e766f651c1f0cd7ed1b0cc9f4072ff3be46246d67"}
Sep 12 02:52:52.294468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.293526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:52:52.306632 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.306546    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf" podStartSLOduration=1.306526039 podStartE2EDuration="1.306526039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:52.304462801 +0000 UTC m=+990.587376380" watchObservedRunningTime="2026-09-12 02:52:52.306526039 +0000 UTC m=+990.589439621"
Sep 12 02:52:52.320879 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.320832    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh" podStartSLOduration=1.3208186450000001 podStartE2EDuration="1.320818645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:52:52.319650478 +0000 UTC m=+990.602564061" watchObservedRunningTime="2026-09-12 02:52:52.320818645 +0000 UTC m=+990.603732280"
Sep 12 02:52:52.325405 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.325383    2579 patch_prober.go:28] interesting pod/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 12 02:52:52.325504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.325429    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 12 02:52:52.786196 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:52.786139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:53.325652 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:53.325618    2579 patch_prober.go:28] interesting pod/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 12 02:52:53.326019 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:53.325662    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 12 02:52:53.776484 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:53.776426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:54.325376 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:54.325348    2579 patch_prober.go:28] interesting pod/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 12 02:52:54.325581 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:54.325399    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 12 02:52:54.479586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:54.479503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:55.178095 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:55.178053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:55.325954 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:55.325917    2579 patch_prober.go:28] interesting pod/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 12 02:52:55.326135 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:55.325984    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 12 02:52:56.085745 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:56.085708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:56.435110 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:56.435079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:56.435598 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:56.435571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:56.436782 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:56.436759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:52:56.850840 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:56.850753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:57.580330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:57.580297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:57.900054 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:57.900024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:58.000424 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:58.000393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:52:58.446040 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:52:58.446011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:01.652145 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:01.652082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:02.211842 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.211809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:02.232742 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.232710    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"]
Sep 12 02:53:02.232994 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.232966    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64" podUID="b949eb71-540c-450c-92aa-3ebea5df46af" containerName="payload-processing" containerID="cri-o://711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2" gracePeriod=30
Sep 12 02:53:02.272466 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.272424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:02.372529 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.372498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:02.464687 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.463626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:02.529914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.523156    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:53:02.576020 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.575978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:02.612246 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.612212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl5l8\" (UniqueName: \"kubernetes.io/projected/b949eb71-540c-450c-92aa-3ebea5df46af-kube-api-access-rl5l8\") pod \"b949eb71-540c-450c-92aa-3ebea5df46af\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") "
Sep 12 02:53:02.612420 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.612276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949eb71-540c-450c-92aa-3ebea5df46af-plugins-config-volume\") pod \"b949eb71-540c-450c-92aa-3ebea5df46af\" (UID: \"b949eb71-540c-450c-92aa-3ebea5df46af\") "
Sep 12 02:53:02.612687 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.612662    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b949eb71-540c-450c-92aa-3ebea5df46af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b949eb71-540c-450c-92aa-3ebea5df46af" (UID: "b949eb71-540c-450c-92aa-3ebea5df46af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:53:02.614463 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.614417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b949eb71-540c-450c-92aa-3ebea5df46af-kube-api-access-rl5l8" (OuterVolumeSpecName: "kube-api-access-rl5l8") pod "b949eb71-540c-450c-92aa-3ebea5df46af" (UID: "b949eb71-540c-450c-92aa-3ebea5df46af"). InnerVolumeSpecName "kube-api-access-rl5l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:53:02.712933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.712890    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949eb71-540c-450c-92aa-3ebea5df46af-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:53:02.712933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.712924    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5l8\" (UniqueName: \"kubernetes.io/projected/b949eb71-540c-450c-92aa-3ebea5df46af-kube-api-access-rl5l8\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:53:02.921872 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:02.921842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:03.059051 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.059007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:03.154918 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.154889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:03.301471 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.301357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 02:53:03.301628 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.301493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 02:53:03.311834 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.311808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:03.357915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.357868    2579 generic.go:356] "Generic (PLEG): container finished" podID="b949eb71-540c-450c-92aa-3ebea5df46af" containerID="711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2" exitCode=0
Sep 12 02:53:03.358098 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.357932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64" event={"ID":"b949eb71-540c-450c-92aa-3ebea5df46af","Type":"ContainerDied","Data":"711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2"}
Sep 12 02:53:03.358098 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.357971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"
Sep 12 02:53:03.358098 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.357983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64" event={"ID":"b949eb71-540c-450c-92aa-3ebea5df46af","Type":"ContainerDied","Data":"7e0bcc903595b58d8e28f01bb06d7376a43aaa3373525c91b34fe9069d76f526"}
Sep 12 02:53:03.358098 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.358017    2579 scope.go:117] "RemoveContainer" containerID="711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2"
Sep 12 02:53:03.372965 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.372919    2579 scope.go:117] "RemoveContainer" containerID="711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2"
Sep 12 02:53:03.373334 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:53:03.373307    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2\": container with ID starting with 711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2 not found: ID does not exist" containerID="711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2"
Sep 12 02:53:03.373418 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.373347    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2"} err="failed to get container status \"711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2\": rpc error: code = NotFound desc = could not find container \"711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2\": container with ID starting with 711be51ef4fd673adde6b496e19355cfeb616f0d29625d60869a520f5ebf0ec2 not found: ID does not exist"
Sep 12 02:53:03.384901 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.384874    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"]
Sep 12 02:53:03.387850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.387827    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-214c0e42-59b6f95cf7-hwb64"]
Sep 12 02:53:03.633605 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.633568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:03.698990 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.698955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:03.798608 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:03.798573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:04.247567 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:04.247530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b949eb71-540c-450c-92aa-3ebea5df46af" path="/var/lib/kubelet/pods/b949eb71-540c-450c-92aa-3ebea5df46af/volumes"
Sep 12 02:53:04.558176 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:04.558087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:05.260788 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.260732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:05.372994 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.372960    2579 generic.go:356] "Generic (PLEG): container finished" podID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerID="077dceae8828c79bdaaf46d5536e900aa0fdf313f4a940d81754c4dff448db6d" exitCode=0
Sep 12 02:53:05.373158 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.373040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" event={"ID":"b57f3ab4-773a-4f5b-a706-bb0e272868b3","Type":"ContainerDied","Data":"077dceae8828c79bdaaf46d5536e900aa0fdf313f4a940d81754c4dff448db6d"}
Sep 12 02:53:05.373489 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.373465    2579 scope.go:117] "RemoveContainer" containerID="077dceae8828c79bdaaf46d5536e900aa0fdf313f4a940d81754c4dff448db6d"
Sep 12 02:53:05.374554 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.374527    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:53:05.374624 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.374598    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:53:05.374668 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:05.374629    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:53:06.013081 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:06.013047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:06.384207 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:06.384163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" event={"ID":"b57f3ab4-773a-4f5b-a706-bb0e272868b3","Type":"ContainerStarted","Data":"95d287196d4d656d6e237bbaecc30661f7e972fd920e12686f3ba5c8709c4771"}
Sep 12 02:53:06.699008 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:06.698919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:53:06.699279 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:06.699247    2579 patch_prober.go:28] interesting pod/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 12 02:53:06.699336 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:06.699312    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 12 02:53:06.946880 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:06.946852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:07.082372 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:07.082268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:07.166837 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:07.166800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:07.699380 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:07.699346    2579 patch_prober.go:28] interesting pod/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 12 02:53:07.699806 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:07.699395    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 12 02:53:08.149975 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.149945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:08.247313 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.247282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:08.348663 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.348629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:08.633670 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.633641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:08.699641 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.699610    2579 patch_prober.go:28] interesting pod/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 12 02:53:08.700091 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.699662    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 12 02:53:08.732064 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.732029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:08.824829 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.824800    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"]
Sep 12 02:53:08.825233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.825220    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b949eb71-540c-450c-92aa-3ebea5df46af" containerName="payload-processing"
Sep 12 02:53:08.825280 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.825235    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b949eb71-540c-450c-92aa-3ebea5df46af" containerName="payload-processing"
Sep 12 02:53:08.825315 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.825297    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b949eb71-540c-450c-92aa-3ebea5df46af" containerName="payload-processing"
Sep 12 02:53:08.828459 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.828430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:08.831008 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.830982    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-03cd9838\""
Sep 12 02:53:08.831263 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.831246    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-03cd9838-dockercfg-fgpgr\""
Sep 12 02:53:08.838277 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.838251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"]
Sep 12 02:53:08.883504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.883471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gxc\" (UniqueName: \"kubernetes.io/projected/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-kube-api-access-g2gxc\") pod \"payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:08.883940 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.883822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:08.990876 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.990793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gxc\" (UniqueName: \"kubernetes.io/projected/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-kube-api-access-g2gxc\") pod \"payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:08.991093 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.990890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:08.991890 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.991862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:08.998965 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:08.998946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gxc\" (UniqueName: \"kubernetes.io/projected/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-kube-api-access-g2gxc\") pod \"payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:09.084942 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:09.084802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:09.140989 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:09.140902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:09.440479 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:09.440439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"]
Sep 12 02:53:09.442242 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:53:09.442216    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a0d29a_5cf4_4e1f_a655_4d5135a07e3f.slice/crio-f5029aa5a46e7cf5de5bcd259654492bbc2958fdd9ee52ec33c0bbdc0fad97ef WatchSource:0}: Error finding container f5029aa5a46e7cf5de5bcd259654492bbc2958fdd9ee52ec33c0bbdc0fad97ef: Status 404 returned error can't find the container with id f5029aa5a46e7cf5de5bcd259654492bbc2958fdd9ee52ec33c0bbdc0fad97ef
Sep 12 02:53:09.698994 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:09.698902    2579 patch_prober.go:28] interesting pod/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 12 02:53:09.698994 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:09.698958    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 12 02:53:09.871996 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:09.871956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:10.335784 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.335750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:10.408209 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.408173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj" event={"ID":"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f","Type":"ContainerStarted","Data":"6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74"}
Sep 12 02:53:10.408403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.408234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj" event={"ID":"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f","Type":"ContainerStarted","Data":"f5029aa5a46e7cf5de5bcd259654492bbc2958fdd9ee52ec33c0bbdc0fad97ef"}
Sep 12 02:53:10.408765 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.408417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:10.424344 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.424294    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj" podStartSLOduration=2.424277669 podStartE2EDuration="2.424277669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:53:10.422379494 +0000 UTC m=+1008.705293077" watchObservedRunningTime="2026-09-12 02:53:10.424277669 +0000 UTC m=+1008.707191285"
Sep 12 02:53:10.698938 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.698900    2579 patch_prober.go:28] interesting pod/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 12 02:53:10.699143 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.698953    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 12 02:53:10.779865 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:10.779678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:11.201345 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:11.201303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:11.569652 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:11.569561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:11.837349 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:11.837266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:53:11.837811 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:11.837785    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:53:11.838869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:11.838841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 02:53:12.228203 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:12.228118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:12.977869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:12.977834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:13.631882 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:13.631849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:21.415868 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:21.415839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:30.130330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:30.130297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:30.830125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:30.830077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:31.460915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:31.460885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:32.197427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:32.197393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:32.954392 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:32.954357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:33.591061 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:33.591027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:34.333172 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:34.333131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:35.004376 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:35.004341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:35.670980 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:35.670949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:36.446264 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:36.446214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:37.482977 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:37.482945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.030049 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.030022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.057852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.057825    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"]
Sep 12 02:53:38.058069 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.058046    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj" podUID="38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" containerName="payload-pre-processing" containerID="cri-o://6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74" gracePeriod=30
Sep 12 02:53:38.093175 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.093146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.193258 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.193223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.280694 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.280602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.334759 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.334734    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:38.377369 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.377336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.486553 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.486524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2gxc\" (UniqueName: \"kubernetes.io/projected/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-kube-api-access-g2gxc\") pod \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") "
Sep 12 02:53:38.487005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.486572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-plugins-config-volume\") pod \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\" (UID: \"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f\") "
Sep 12 02:53:38.487005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.486984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" (UID: "38a0d29a-5cf4-4e1f-a655-4d5135a07e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:53:38.489017 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.488988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-kube-api-access-g2gxc" (OuterVolumeSpecName: "kube-api-access-g2gxc") pod "38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" (UID: "38a0d29a-5cf4-4e1f-a655-4d5135a07e3f"). InnerVolumeSpecName "kube-api-access-g2gxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:53:38.553559 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.553480    2579 generic.go:356] "Generic (PLEG): container finished" podID="38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" containerID="6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74" exitCode=0
Sep 12 02:53:38.553559 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.553528    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"
Sep 12 02:53:38.553740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.553564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj" event={"ID":"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f","Type":"ContainerDied","Data":"6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74"}
Sep 12 02:53:38.553740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.553620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj" event={"ID":"38a0d29a-5cf4-4e1f-a655-4d5135a07e3f","Type":"ContainerDied","Data":"f5029aa5a46e7cf5de5bcd259654492bbc2958fdd9ee52ec33c0bbdc0fad97ef"}
Sep 12 02:53:38.553740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.553643    2579 scope.go:117] "RemoveContainer" containerID="6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74"
Sep 12 02:53:38.563146 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.563121    2579 scope.go:117] "RemoveContainer" containerID="6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74"
Sep 12 02:53:38.563537 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:53:38.563517    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74\": container with ID starting with 6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74 not found: ID does not exist" containerID="6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74"
Sep 12 02:53:38.563646 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.563545    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74"} err="failed to get container status \"6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74\": rpc error: code = NotFound desc = could not find container \"6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74\": container with ID starting with 6f91ac53e10886807fee12d71aa8890f8225451bb8b94fa593cfad7d4779ae74 not found: ID does not exist"
Sep 12 02:53:38.574591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.574565    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"]
Sep 12 02:53:38.576437 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.576420    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-03cd9838-7f49b9c86-wqlcj"]
Sep 12 02:53:38.587782 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.587758    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2gxc\" (UniqueName: \"kubernetes.io/projected/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-kube-api-access-g2gxc\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:53:38.587782 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.587780    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:53:38.671365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.671325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:38.837835 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:38.837744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:39.137254 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:39.137224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:39.317749 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:39.317714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:39.509716 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:39.509634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:40.245880 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:40.245853    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" path="/var/lib/kubelet/pods/38a0d29a-5cf4-4e1f-a655-4d5135a07e3f/volumes"
Sep 12 02:53:40.246143 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:40.246121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:40.981959 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:40.981925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:41.648664 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:41.648615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:41.755972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:41.755937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:41.856220 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:41.856184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:42.446140 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:42.446109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:43.232033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:43.232005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:43.330352 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:43.330321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:43.458992 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:43.458951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:43.764799 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:43.764771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:43.867945 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:43.867910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:44.161527 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:44.161494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:44.929238 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:44.929206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:45.862032 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:45.861994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:46.619470 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:46.619414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:48.414021 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:48.413988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:48.466462 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:48.466430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:53:58.619583 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:53:58.619545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:05.239671 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:05.239639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:05.986257 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:05.986219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:06.682383 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:06.682350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:07.448660 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:07.448622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:08.143251 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:08.143199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:08.843091 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:08.843054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:09.469093 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:09.469059    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:10.232931 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:10.232894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:10.860159 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:10.860122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:11.530874 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:11.530834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:12.262151 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:12.262118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:13.057311 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:13.057224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:13.593422 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:13.593396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:13.660655 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:13.660629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:13.722850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:13.722795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:13.789965 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:13.789937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:13.856783 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:13.856759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:14.170586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:14.170505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:14.944791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:14.944763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:15.031022 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:15.030991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:15.486468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:15.486432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:15.659703 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:15.659670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:15.807397 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:15.807308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:16.183211 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:16.183177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:16.339926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:16.339892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:17.169567 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:17.169537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:17.927813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:17.927778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:18.150115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:18.150080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:18.258396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:18.258315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:18.661514 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:18.661472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:19.537518 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:19.537473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:20.161838 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:20.161801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:20.797313 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:20.797285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:21.513475 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:21.513412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:22.273526 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:22.273483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:22.372806 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:22.372772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:22.536201 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:22.536121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:22.788143 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:22.788058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:22.891146 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:22.891116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:27.805572 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:27.805535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:41.739083 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:41.739045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:44.278553 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:44.278504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:45.133994 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:45.133959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:45.768887 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:45.768849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:46.066273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:46.066180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:46.408702 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:46.408630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:47.161410 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:47.161369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:47.841389 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:47.841354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:48.705924 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:48.705889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:49.548669 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:49.548641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:50.205342 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:50.205305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:50.881701 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:50.881661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:51.688581 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:51.688531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:52.420332 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:52.420275    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:53.206750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:53.206713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:55.641428 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:55.641397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:54:59.526805 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:54:59.526773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:00.411995 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:00.411962    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:01.234694 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:01.234664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:01.940062 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:01.940027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:02.656339 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:02.656303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:03.375519 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:03.375481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:04.131853 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:04.131801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:04.720984 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:04.720953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:04.800943 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:04.800913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:04.874369 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:04.874334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:04.941598 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:04.941561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:05.070828 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:05.070727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:05.311313 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:05.311282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:05.761356 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:05.761317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:05.856217 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:05.856186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:05.958024 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:05.957992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:06.275510 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:06.275469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:06.972703 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:06.972670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:07.379869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:07.379837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:07.789113 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:07.789019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:08.452850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:08.452757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:09.248757 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:09.248726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:10.209789 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:10.209754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:10.785618 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:10.785591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:10.876785 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:10.876743    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:10.953437 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:10.953404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:11.026204 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:11.026170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:11.089660 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:11.089562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:11.380645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:11.380612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:11.480195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:11.480165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:12.164782 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:12.164750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:12.958938 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:12.958887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:13.602574 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:13.602537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:13.671882 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:13.671837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:13.765149 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:13.765115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:13.869187 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:13.869153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:13.965275 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:13.965218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:14.346439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:14.346348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:14.666334 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:14.666301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:16.458580 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:16.458550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:17.323234 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:17.323202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:19.822026 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:19.821995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:37.159972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:37.159936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:37.389945 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:37.389910    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"]
Sep 12 02:55:37.390157 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:37.390134    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" containerID="cri-o://3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76" gracePeriod=30
Sep 12 02:55:38.519500 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:38.519468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:40.327991 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:40.327927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:40.529545 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:40.529517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:41.151028 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:41.150998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:41.251597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:41.251562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:41.437765 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:41.437672    2579 patch_prober.go:28] interesting pod/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:55:41.438172 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:41.437766    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:55:42.155708 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.155680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:42.655553 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.655531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:55:42.769661 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769586    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-token\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769661 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvqll\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-kube-api-access-cvqll\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-envoy\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-certs\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769720    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-data\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-socket\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-credential-socket\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-podinfo\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.769881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istiod-ca-cert\") pod \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\" (UID: \"9d0f90e8-ca5c-40e4-99be-ac759aa73777\") "
Sep 12 02:55:42.770368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.770007    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:42.770368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.769991    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:42.770368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.770156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:42.770368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.770176    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.770368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.770174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:42.770368 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.770192    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.771137 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.770491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:55:42.772048 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.772023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:55:42.772217 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.772198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:55:42.772297 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.772202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:55:42.772297 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.772273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-kube-api-access-cvqll" (OuterVolumeSpecName: "kube-api-access-cvqll") pod "9d0f90e8-ca5c-40e4-99be-ac759aa73777" (UID: "9d0f90e8-ca5c-40e4-99be-ac759aa73777"). InnerVolumeSpecName "kube-api-access-cvqll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:55:42.871035 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871006    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.871035 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871033    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.871206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871042    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqll\" (UniqueName: \"kubernetes.io/projected/9d0f90e8-ca5c-40e4-99be-ac759aa73777-kube-api-access-cvqll\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.871206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871051    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.871206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871059    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.871206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871068    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0f90e8-ca5c-40e4-99be-ac759aa73777-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.871206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.871076    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0f90e8-ca5c-40e4-99be-ac759aa73777-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:55:42.936780 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:42.936749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:43.186628 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.186598    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerID="3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76" exitCode=0
Sep 12 02:55:43.186791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.186639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" event={"ID":"9d0f90e8-ca5c-40e4-99be-ac759aa73777","Type":"ContainerDied","Data":"3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76"}
Sep 12 02:55:43.186791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.186648    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"
Sep 12 02:55:43.186791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.186681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv" event={"ID":"9d0f90e8-ca5c-40e4-99be-ac759aa73777","Type":"ContainerDied","Data":"be93889bfd80d834e1d28c05a79e23cc6be9db3d6ae59eb27829d7bd50a32053"}
Sep 12 02:55:43.186791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.186702    2579 scope.go:117] "RemoveContainer" containerID="3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76"
Sep 12 02:55:43.196429 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.196414    2579 scope.go:117] "RemoveContainer" containerID="63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"
Sep 12 02:55:43.204389 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.204373    2579 scope.go:117] "RemoveContainer" containerID="3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76"
Sep 12 02:55:43.204758 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:55:43.204739    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76\": container with ID starting with 3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76 not found: ID does not exist" containerID="3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76"
Sep 12 02:55:43.204858 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.204768    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76"} err="failed to get container status \"3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76\": rpc error: code = NotFound desc = could not find container \"3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76\": container with ID starting with 3b2df87f15e2da94cea809bdd82f7bedd9bdabe039d1c5866adfd906dc51cd76 not found: ID does not exist"
Sep 12 02:55:43.204858 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.204788    2579 scope.go:117] "RemoveContainer" containerID="63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"
Sep 12 02:55:43.205060 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:55:43.205041    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781\": container with ID starting with 63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781 not found: ID does not exist" containerID="63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"
Sep 12 02:55:43.205099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.205067    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781"} err="failed to get container status \"63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781\": rpc error: code = NotFound desc = could not find container \"63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781\": container with ID starting with 63e43e201d61d9ab85d85a0d38023769ffaa299c37c455c81c8a8f48c5f13781 not found: ID does not exist"
Sep 12 02:55:43.211099 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.211078    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"]
Sep 12 02:55:43.215967 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.215935    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4b3be7-openshift-default-5dc84455d4-4czvv"]
Sep 12 02:55:43.727850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:43.727818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:44.245432 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:44.245403    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" path="/var/lib/kubelet/pods/9d0f90e8-ca5c-40e4-99be-ac759aa73777/volumes"
Sep 12 02:55:44.452468 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:44.452418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:44.515633 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:44.515527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:44.622560 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:44.622505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:44.923755 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:44.923724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:45.005679 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:45.005652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:46.313617 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:46.313579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:46.421356 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:46.421316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:46.512252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:46.512219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:55:54.424347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:55:54.424315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:02.408545 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:02.408486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:06.166151 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:06.166105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:06.478087 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:06.477992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:07.277002 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:07.276967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:08.035041 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:08.034998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:08.375618 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:08.375587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:08.483877 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:08.483836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:08.576299 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:08.576247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:08.888773 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:08.888723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:09.765576 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:09.765546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:10.452610 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:10.452570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:10.609699 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:10.609657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:18.542886 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:18.542853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:19.097728 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:19.097697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:19.955303 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:19.955255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:22.210656 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:22.210631    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:56:22.211340 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:22.211322    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 02:56:25.819272 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:25.819237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:25.883901 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:25.883870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:25.965877 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:25.965834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:26.065749 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:26.065686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:26.168536 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:26.168508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:26.827902 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:26.827860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:26.926191 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:26.926141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:27.026986 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:27.026959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:28.600854 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:28.600825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:34.422201 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:34.422161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:34.522396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:34.522362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:35.116539 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:35.116500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:41.098697 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:41.098648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:41.201190 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:41.201135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:41.357776 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:41.357747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:41.697944 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:41.697840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:41.946181 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:41.946148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:46.938521 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:46.938483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:47.643780 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:47.643746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:47.739819 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:47.739787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:48.657488 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:48.657416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:49.415739 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:49.415708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:50.125001 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:50.124969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:50.867153 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:50.867120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:51.632428 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:51.632392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:51.790822 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:51.790783    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:51.891548 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:51.891434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:52.171745 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.171669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:52.269366 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.269331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:56:52.281034 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"]
Sep 12 02:56:52.281382 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281371    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy"
Sep 12 02:56:52.281427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281384    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy"
Sep 12 02:56:52.281427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281410    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" containerName="payload-pre-processing"
Sep 12 02:56:52.281427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281415    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" containerName="payload-pre-processing"
Sep 12 02:56:52.281545 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281486    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a0d29a-5cf4-4e1f-a655-4d5135a07e3f" containerName="payload-pre-processing"
Sep 12 02:56:52.281545 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281496    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy"
Sep 12 02:56:52.281545 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.281503    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy"
Sep 12 02:56:52.284481 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.284442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.287170 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.287149    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-662d15cb\""
Sep 12 02:56:52.287314 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.287270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-662d15cb-dockercfg-jvx67\""
Sep 12 02:56:52.293979 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.293565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"]
Sep 12 02:56:52.296376 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.296355    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"]
Sep 12 02:56:52.296965 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.296946    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy"
Sep 12 02:56:52.297033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.296970    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0f90e8-ca5c-40e4-99be-ac759aa73777" containerName="istio-proxy"
Sep 12 02:56:52.300287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.300270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.311152 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.311127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"]
Sep 12 02:56:52.430769 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.430689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rxl\" (UniqueName: \"kubernetes.io/projected/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-kube-api-access-m5rxl\") pod \"payload-processing-e2e-mt-662d15cb-6cf459c997-8646k\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.430917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.430765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0000dd-8364-459a-b95f-44d92d7de354-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.430917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.430798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-plugins-config-volume\") pod \"payload-processing-e2e-mt-662d15cb-6cf459c997-8646k\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.430917 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.430825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfcv\" (UniqueName: \"kubernetes.io/projected/5d0000dd-8364-459a-b95f-44d92d7de354-kube-api-access-8jfcv\") pod \"payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.532225 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.532194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rxl\" (UniqueName: \"kubernetes.io/projected/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-kube-api-access-m5rxl\") pod \"payload-processing-e2e-mt-662d15cb-6cf459c997-8646k\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.532396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.532255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0000dd-8364-459a-b95f-44d92d7de354-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.532396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.532291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-plugins-config-volume\") pod \"payload-processing-e2e-mt-662d15cb-6cf459c997-8646k\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.532396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.532315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfcv\" (UniqueName: \"kubernetes.io/projected/5d0000dd-8364-459a-b95f-44d92d7de354-kube-api-access-8jfcv\") pod \"payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.533037 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.533015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0000dd-8364-459a-b95f-44d92d7de354-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.533037 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.533031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-plugins-config-volume\") pod \"payload-processing-e2e-mt-662d15cb-6cf459c997-8646k\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.541506 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.541485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rxl\" (UniqueName: \"kubernetes.io/projected/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-kube-api-access-m5rxl\") pod \"payload-processing-e2e-mt-662d15cb-6cf459c997-8646k\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") " pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.541598 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.541554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfcv\" (UniqueName: \"kubernetes.io/projected/5d0000dd-8364-459a-b95f-44d92d7de354-kube-api-access-8jfcv\") pod \"payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.596919 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.596873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:52.613324 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.613260    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:52.873895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.873860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"]
Sep 12 02:56:52.876863 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:56:52.876836    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0000dd_8364_459a_b95f_44d92d7de354.slice/crio-f1829ed8be7e289f1c7c39387337b3c1c443da695bc929b6affc41f706eec9c3 WatchSource:0}: Error finding container f1829ed8be7e289f1c7c39387337b3c1c443da695bc929b6affc41f706eec9c3: Status 404 returned error can't find the container with id f1829ed8be7e289f1c7c39387337b3c1c443da695bc929b6affc41f706eec9c3
Sep 12 02:56:52.898885 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:52.898861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"]
Sep 12 02:56:52.899530 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:56:52.899510    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eecfe3d_d0a7_4ea0_8217_67ca04e92ad0.slice/crio-6cb583061b8565e4666738e6a7480bf6c4a16f069453fc3babb0637214ded0c2 WatchSource:0}: Error finding container 6cb583061b8565e4666738e6a7480bf6c4a16f069453fc3babb0637214ded0c2: Status 404 returned error can't find the container with id 6cb583061b8565e4666738e6a7480bf6c4a16f069453fc3babb0637214ded0c2
Sep 12 02:56:53.538144 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.538114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k" event={"ID":"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0","Type":"ContainerStarted","Data":"1412bde79dfa71ff7201e5b78253386d725cc805dddb7511d0639ad6cda69f87"}
Sep 12 02:56:53.538352 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.538163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k" event={"ID":"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0","Type":"ContainerStarted","Data":"6cb583061b8565e4666738e6a7480bf6c4a16f069453fc3babb0637214ded0c2"}
Sep 12 02:56:53.538352 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.538220    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:56:53.539892 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.539866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw" event={"ID":"5d0000dd-8364-459a-b95f-44d92d7de354","Type":"ContainerStarted","Data":"c1fb44ffad1c7c50810dcfa49bbedfc0a0ed1eebb1834119855795c71d5dff8d"}
Sep 12 02:56:53.540033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.539905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw" event={"ID":"5d0000dd-8364-459a-b95f-44d92d7de354","Type":"ContainerStarted","Data":"f1829ed8be7e289f1c7c39387337b3c1c443da695bc929b6affc41f706eec9c3"}
Sep 12 02:56:53.540033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.539929    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:56:53.555519 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.555475    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k" podStartSLOduration=1.5554612890000001 podStartE2EDuration="1.555461289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:56:53.554112864 +0000 UTC m=+1231.837026449" watchObservedRunningTime="2026-09-12 02:56:53.555461289 +0000 UTC m=+1231.838374861"
Sep 12 02:56:53.571763 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:56:53.571716    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw" podStartSLOduration=1.571699771 podStartE2EDuration="1.571699771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:56:53.569815559 +0000 UTC m=+1231.852729143" watchObservedRunningTime="2026-09-12 02:56:53.571699771 +0000 UTC m=+1231.854613402"
Sep 12 02:57:04.547016 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:04.546986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:57:04.547475 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:04.547057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:57:13.729684 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:13.729604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:14.672922 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:14.672892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:15.347627 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:15.347600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:16.060632 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:16.060591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:16.852636 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:16.852607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:17.428065 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.428029    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"]
Sep 12 02:57:17.432643 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.432621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.437115 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.437096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-450f2a\"/\"default-dockercfg-wzvhb\""
Sep 12 02:57:17.437339 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.437320    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-450f2a\"/\"kube-root-ca.crt\""
Sep 12 02:57:17.437473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.437386    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-450f2a\"/\"openshift-service-ca.crt\""
Sep 12 02:57:17.437569 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.437545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-450f2a\"/\"explicit-model-450f2a-kserve-self-signed-certs\""
Sep 12 02:57:17.440825 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.440812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"]
Sep 12 02:57:17.457229 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.457205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-model-cache\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.457334 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.457236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.457334 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.457297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-home\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.457334 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.457322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tmp-dir\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.457463 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.457417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-dshm\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.457518 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.457498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zv2j\" (UniqueName: \"kubernetes.io/projected/6c259f24-bb5f-4a7a-9891-62422eaabe1a-kube-api-access-5zv2j\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.465050 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.465028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:17.558197 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-dshm\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558197 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zv2j\" (UniqueName: \"kubernetes.io/projected/6c259f24-bb5f-4a7a-9891-62422eaabe1a-kube-api-access-5zv2j\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-model-cache\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558534 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-home\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558534 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tmp-dir\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558763 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-model-cache\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558847 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tmp-dir\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.558847 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.558815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-home\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.560656 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.560626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-dshm\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.561030 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.561011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.567529 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.567509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zv2j\" (UniqueName: \"kubernetes.io/projected/6c259f24-bb5f-4a7a-9891-62422eaabe1a-kube-api-access-5zv2j\") pod \"explicit-model-450f2a-kserve-6495b6697c-jlnr4\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") " pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.743597 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.743505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:17.755843 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.755819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:17.852959 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:17.852928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:18.016740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.016718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"]
Sep 12 02:57:18.018276 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:57:18.018249    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c259f24_bb5f_4a7a_9891_62422eaabe1a.slice/crio-b995b406780ffc59e1b7c1c6ddd7e793c50084107fb648ff83ace3b683f8bc7f WatchSource:0}: Error finding container b995b406780ffc59e1b7c1c6ddd7e793c50084107fb648ff83ace3b683f8bc7f: Status 404 returned error can't find the container with id b995b406780ffc59e1b7c1c6ddd7e793c50084107fb648ff83ace3b683f8bc7f
Sep 12 02:57:18.452056 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.452021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:18.561814 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.561763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:18.674936 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.674907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4" event={"ID":"6c259f24-bb5f-4a7a-9891-62422eaabe1a","Type":"ContainerStarted","Data":"0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8"}
Sep 12 02:57:18.675118 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.674949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4" event={"ID":"6c259f24-bb5f-4a7a-9891-62422eaabe1a","Type":"ContainerStarted","Data":"b995b406780ffc59e1b7c1c6ddd7e793c50084107fb648ff83ace3b683f8bc7f"}
Sep 12 02:57:18.704423 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.704325    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4" podStartSLOduration=1.704308479 podStartE2EDuration="1.704308479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:57:18.701884193 +0000 UTC m=+1256.984797774" watchObservedRunningTime="2026-09-12 02:57:18.704308479 +0000 UTC m=+1256.987222094"
Sep 12 02:57:18.922100 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:18.922057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:24.873937 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:24.873904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:24.974187 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:24.974158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:26.071439 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:26.071409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:26.862168 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:26.862138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:27.652777 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:27.652735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:27.744035 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:27.744001    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:27.744238 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:27.744046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:27.756996 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:27.756973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:28.471330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:28.471302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:28.740400 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:28.740314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:29.225085 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:29.225051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:29.878350 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:29.878315    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-self-signed-certs: secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:29.878858 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:29.878406    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs podName:6c259f24-bb5f-4a7a-9891-62422eaabe1a nodeName:}" failed. No retries permitted until 2026-09-12 02:57:30.378382771 +0000 UTC m=+1268.661296338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs") pod "explicit-model-450f2a-kserve-6495b6697c-jlnr4" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a") : secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:29.879406 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:29.879380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"]
Sep 12 02:57:29.896491 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:29.896438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:30.000750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:30.000714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:30.383564 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:30.383536    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-self-signed-certs: secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:30.383727 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:30.383601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs podName:6c259f24-bb5f-4a7a-9891-62422eaabe1a nodeName:}" failed. No retries permitted until 2026-09-12 02:57:31.383582796 +0000 UTC m=+1269.666496354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs") pod "explicit-model-450f2a-kserve-6495b6697c-jlnr4" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a") : secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:30.675215 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:30.675143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:31.394066 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:31.394031    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-self-signed-certs: secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:31.394481 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:31.394099    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs podName:6c259f24-bb5f-4a7a-9891-62422eaabe1a nodeName:}" failed. No retries permitted until 2026-09-12 02:57:33.394085496 +0000 UTC m=+1271.676999056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs") pod "explicit-model-450f2a-kserve-6495b6697c-jlnr4" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a") : secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:31.567077 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:31.567049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:32.347942 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:32.347904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:33.241385 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:33.241352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:33.414661 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:33.414630    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-self-signed-certs: secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:33.414833 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:33.414706    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs podName:6c259f24-bb5f-4a7a-9891-62422eaabe1a nodeName:}" failed. No retries permitted until 2026-09-12 02:57:37.414687498 +0000 UTC m=+1275.697601056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs") pod "explicit-model-450f2a-kserve-6495b6697c-jlnr4" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a") : secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:34.006947 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:34.006920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:34.846745 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:34.846712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:35.755531 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:35.755495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:36.516048 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.516016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:36.556262 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.556231    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"]
Sep 12 02:57:36.556529 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.556500    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k" podUID="1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" containerName="payload-processing" containerID="cri-o://1412bde79dfa71ff7201e5b78253386d725cc805dddb7511d0639ad6cda69f87" gracePeriod=30
Sep 12 02:57:36.569591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.569202    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"]
Sep 12 02:57:36.569591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.569418    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw" podUID="5d0000dd-8364-459a-b95f-44d92d7de354" containerName="payload-pre-processing" containerID="cri-o://c1fb44ffad1c7c50810dcfa49bbedfc0a0ed1eebb1834119855795c71d5dff8d" gracePeriod=30
Sep 12 02:57:36.605121 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.605096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:36.704671 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.704638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:36.772902 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.772193    2579 generic.go:356] "Generic (PLEG): container finished" podID="1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" containerID="1412bde79dfa71ff7201e5b78253386d725cc805dddb7511d0639ad6cda69f87" exitCode=0
Sep 12 02:57:36.772902 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.772307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k" event={"ID":"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0","Type":"ContainerDied","Data":"1412bde79dfa71ff7201e5b78253386d725cc805dddb7511d0639ad6cda69f87"}
Sep 12 02:57:36.774752 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.774660    2579 generic.go:356] "Generic (PLEG): container finished" podID="5d0000dd-8364-459a-b95f-44d92d7de354" containerID="c1fb44ffad1c7c50810dcfa49bbedfc0a0ed1eebb1834119855795c71d5dff8d" exitCode=0
Sep 12 02:57:36.774834 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.774801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw" event={"ID":"5d0000dd-8364-459a-b95f-44d92d7de354","Type":"ContainerDied","Data":"c1fb44ffad1c7c50810dcfa49bbedfc0a0ed1eebb1834119855795c71d5dff8d"}
Sep 12 02:57:36.800415 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.800386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:36.853618 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.853581    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:57:36.856824 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.856802    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:57:36.948767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.948737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jfcv\" (UniqueName: \"kubernetes.io/projected/5d0000dd-8364-459a-b95f-44d92d7de354-kube-api-access-8jfcv\") pod \"5d0000dd-8364-459a-b95f-44d92d7de354\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") "
Sep 12 02:57:36.949005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.948788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5rxl\" (UniqueName: \"kubernetes.io/projected/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-kube-api-access-m5rxl\") pod \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") "
Sep 12 02:57:36.949005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.948842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-plugins-config-volume\") pod \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\" (UID: \"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0\") "
Sep 12 02:57:36.949005 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.948962    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0000dd-8364-459a-b95f-44d92d7de354-plugins-config-volume\") pod \"5d0000dd-8364-459a-b95f-44d92d7de354\" (UID: \"5d0000dd-8364-459a-b95f-44d92d7de354\") "
Sep 12 02:57:36.956287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.949549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0000dd-8364-459a-b95f-44d92d7de354-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d0000dd-8364-459a-b95f-44d92d7de354" (UID: "5d0000dd-8364-459a-b95f-44d92d7de354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:57:36.956287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.949574    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" (UID: "1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:57:36.956287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.950155    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0000dd-8364-459a-b95f-44d92d7de354-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:36.956287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.950186    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:36.956287 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.953962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-kube-api-access-m5rxl" (OuterVolumeSpecName: "kube-api-access-m5rxl") pod "1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" (UID: "1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0"). InnerVolumeSpecName "kube-api-access-m5rxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:57:36.956674 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:36.956483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0000dd-8364-459a-b95f-44d92d7de354-kube-api-access-8jfcv" (OuterVolumeSpecName: "kube-api-access-8jfcv") pod "5d0000dd-8364-459a-b95f-44d92d7de354" (UID: "5d0000dd-8364-459a-b95f-44d92d7de354"). InnerVolumeSpecName "kube-api-access-8jfcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:57:37.051662 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.051579    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfcv\" (UniqueName: \"kubernetes.io/projected/5d0000dd-8364-459a-b95f-44d92d7de354-kube-api-access-8jfcv\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:37.051662 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.051607    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5rxl\" (UniqueName: \"kubernetes.io/projected/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0-kube-api-access-m5rxl\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:37.146258 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.146211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:37.285009 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.284958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:37.455836 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:37.455803    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-self-signed-certs: secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:37.456000 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:37.455867    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs podName:6c259f24-bb5f-4a7a-9891-62422eaabe1a nodeName:}" failed. No retries permitted until 2026-09-12 02:57:45.455852704 +0000 UTC m=+1283.738766263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs") pod "explicit-model-450f2a-kserve-6495b6697c-jlnr4" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a") : secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:37.780740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.780665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k" event={"ID":"1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0","Type":"ContainerDied","Data":"6cb583061b8565e4666738e6a7480bf6c4a16f069453fc3babb0637214ded0c2"}
Sep 12 02:57:37.780740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.780727    2579 scope.go:117] "RemoveContainer" containerID="1412bde79dfa71ff7201e5b78253386d725cc805dddb7511d0639ad6cda69f87"
Sep 12 02:57:37.781225 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.780672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"
Sep 12 02:57:37.782155 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.782139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"
Sep 12 02:57:37.782155 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.782159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw" event={"ID":"5d0000dd-8364-459a-b95f-44d92d7de354","Type":"ContainerDied","Data":"f1829ed8be7e289f1c7c39387337b3c1c443da695bc929b6affc41f706eec9c3"}
Sep 12 02:57:37.791371 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.791352    2579 scope.go:117] "RemoveContainer" containerID="c1fb44ffad1c7c50810dcfa49bbedfc0a0ed1eebb1834119855795c71d5dff8d"
Sep 12 02:57:37.807813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.807787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"]
Sep 12 02:57:37.811619 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.811599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-662d15cb-76f84dbdc4-7dprw"]
Sep 12 02:57:37.821654 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.821635    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"]
Sep 12 02:57:37.828108 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:37.828085    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-662d15cb-6cf459c997-8646k"]
Sep 12 02:57:38.031676 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:38.031598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:38.245515 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:38.245484    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" path="/var/lib/kubelet/pods/1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0/volumes"
Sep 12 02:57:38.245914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:38.245899    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d0000dd-8364-459a-b95f-44d92d7de354" path="/var/lib/kubelet/pods/5d0000dd-8364-459a-b95f-44d92d7de354/volumes"
Sep 12 02:57:38.741248 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:38.741223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:39.522166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:39.522131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:40.192125 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:40.192082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:41.025088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:41.025041    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:41.592623 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:41.592598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:41.664849 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:41.664817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:41.714166 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:41.714140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:41.784908 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:41.784869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:41.854056 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:41.853961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:42.077756 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:42.077715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:42.172998 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:42.172969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:42.273895 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:42.273866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:43.026740 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:43.026679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:43.621483 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:43.621426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:43.721414 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:43.721375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:43.847863 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:43.847831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:43.945997 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:43.945910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:44.228557 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:44.228471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:44.340498 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:44.340442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:44.490849 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:44.490746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:44.678675 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:44.678643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:45.450887 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:45.450860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:45.534652 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:45.534622    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-self-signed-certs: secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:45.534802 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:45.534695    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs podName:6c259f24-bb5f-4a7a-9891-62422eaabe1a nodeName:}" failed. No retries permitted until 2026-09-12 02:58:01.534679254 +0000 UTC m=+1299.817592817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs") pod "explicit-model-450f2a-kserve-6495b6697c-jlnr4" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a") : secret "explicit-model-450f2a-kserve-self-signed-certs" not found
Sep 12 02:57:45.760149 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:45.760065    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4" podUID="6c259f24-bb5f-4a7a-9891-62422eaabe1a" containerName="main" containerID="cri-o://0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8" gracePeriod=45
Sep 12 02:57:46.029852 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.029824    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:46.038795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.038753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tmp-dir\") pod \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") "
Sep 12 02:57:46.038795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.038801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zv2j\" (UniqueName: \"kubernetes.io/projected/6c259f24-bb5f-4a7a-9891-62422eaabe1a-kube-api-access-5zv2j\") pod \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") "
Sep 12 02:57:46.039003 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.038858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-home\") pod \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") "
Sep 12 02:57:46.039003 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.038901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-dshm\") pod \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") "
Sep 12 02:57:46.039003 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.038974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-model-cache\") pod \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") "
Sep 12 02:57:46.039160 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039015    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs\") pod \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\" (UID: \"6c259f24-bb5f-4a7a-9891-62422eaabe1a\") "
Sep 12 02:57:46.039160 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c259f24-bb5f-4a7a-9891-62422eaabe1a" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:57:46.039160 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-home" (OuterVolumeSpecName: "home") pod "6c259f24-bb5f-4a7a-9891-62422eaabe1a" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:57:46.039317 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c259f24-bb5f-4a7a-9891-62422eaabe1a" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:57:46.039396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039377    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:46.039470 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039401    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:46.039470 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.039413    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:46.041707 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.041671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-dshm" (OuterVolumeSpecName: "dshm") pod "6c259f24-bb5f-4a7a-9891-62422eaabe1a" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:57:46.041839 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.041752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c259f24-bb5f-4a7a-9891-62422eaabe1a-kube-api-access-5zv2j" (OuterVolumeSpecName: "kube-api-access-5zv2j") pod "6c259f24-bb5f-4a7a-9891-62422eaabe1a" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a"). InnerVolumeSpecName "kube-api-access-5zv2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:57:46.041939 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.041920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c259f24-bb5f-4a7a-9891-62422eaabe1a" (UID: "6c259f24-bb5f-4a7a-9891-62422eaabe1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:57:46.140569 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.140542    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c259f24-bb5f-4a7a-9891-62422eaabe1a-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:46.140569 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.140566    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zv2j\" (UniqueName: \"kubernetes.io/projected/6c259f24-bb5f-4a7a-9891-62422eaabe1a-kube-api-access-5zv2j\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:46.140569 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.140576    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c259f24-bb5f-4a7a-9891-62422eaabe1a-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:57:46.831842 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.831811    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c259f24-bb5f-4a7a-9891-62422eaabe1a" containerID="0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8" exitCode=0
Sep 12 02:57:46.832341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.831853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"
Sep 12 02:57:46.832341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.831904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4" event={"ID":"6c259f24-bb5f-4a7a-9891-62422eaabe1a","Type":"ContainerDied","Data":"0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8"}
Sep 12 02:57:46.832341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.831953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4" event={"ID":"6c259f24-bb5f-4a7a-9891-62422eaabe1a","Type":"ContainerDied","Data":"b995b406780ffc59e1b7c1c6ddd7e793c50084107fb648ff83ace3b683f8bc7f"}
Sep 12 02:57:46.832341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.831978    2579 scope.go:117] "RemoveContainer" containerID="0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8"
Sep 12 02:57:46.840493 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.840474    2579 scope.go:117] "RemoveContainer" containerID="0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8"
Sep 12 02:57:46.840754 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:57:46.840737    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8\": container with ID starting with 0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8 not found: ID does not exist" containerID="0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8"
Sep 12 02:57:46.840814 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.840761    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8"} err="failed to get container status \"0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8\": rpc error: code = NotFound desc = could not find container \"0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8\": container with ID starting with 0f89fd0b323b4ee5ff85940ea3b7d1b2547eb588141fca6340797a7748f0a3a8 not found: ID does not exist"
Sep 12 02:57:46.848230 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.848210    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"]
Sep 12 02:57:46.852524 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:46.852505    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-450f2a/explicit-model-450f2a-kserve-6495b6697c-jlnr4"]
Sep 12 02:57:47.908016 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:47.907984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:48.008696 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:48.008669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:48.106804 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:48.106777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:48.173576 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:48.173502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:48.250001 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:48.249966    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c259f24-bb5f-4a7a-9891-62422eaabe1a" path="/var/lib/kubelet/pods/6c259f24-bb5f-4a7a-9891-62422eaabe1a/volumes"
Sep 12 02:57:48.618553 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:48.618526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:48.976277 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:48.976185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:49.316923 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:49.316825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.127795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.127748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.240504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.240469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.301570 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301541    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"]
Sep 12 02:57:50.301933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0000dd-8364-459a-b95f-44d92d7de354" containerName="payload-pre-processing"
Sep 12 02:57:50.301933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301933    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0000dd-8364-459a-b95f-44d92d7de354" containerName="payload-pre-processing"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301956    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" containerName="payload-processing"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301962    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" containerName="payload-processing"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301971    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c259f24-bb5f-4a7a-9891-62422eaabe1a" containerName="main"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.301976    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c259f24-bb5f-4a7a-9891-62422eaabe1a" containerName="main"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.302028    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d0000dd-8364-459a-b95f-44d92d7de354" containerName="payload-pre-processing"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.302035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eecfe3d-d0a7-4ea0-8217-67ca04e92ad0" containerName="payload-processing"
Sep 12 02:57:50.302088 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.302042    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c259f24-bb5f-4a7a-9891-62422eaabe1a" containerName="main"
Sep 12 02:57:50.309743 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.309717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.313585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.313565    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"openshift-service-ca.crt\""
Sep 12 02:57:50.315092 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.315068    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"kube-root-ca.crt\""
Sep 12 02:57:50.315223 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.315202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"distinct-w0-b0573c-kserve-self-signed-certs\""
Sep 12 02:57:50.315971 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.315953    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"default-dockercfg-lbnv8\""
Sep 12 02:57:50.326484 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.326436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"]
Sep 12 02:57:50.378767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.378679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-tmp-dir\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.378767 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.378727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qr4p\" (UniqueName: \"kubernetes.io/projected/5c90f719-277e-4f8a-b672-046d026cdd3c-kube-api-access-2qr4p\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.378987 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.378794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-dshm\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.378987 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.378822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-home\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.378987 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.378869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-model-cache\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.378987 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.378944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c90f719-277e-4f8a-b672-046d026cdd3c-tls-certs\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.383732 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.383701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.474195 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.474157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.480111 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c90f719-277e-4f8a-b672-046d026cdd3c-tls-certs\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480267 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-tmp-dir\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480267 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qr4p\" (UniqueName: \"kubernetes.io/projected/5c90f719-277e-4f8a-b672-046d026cdd3c-kube-api-access-2qr4p\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-dshm\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-home\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480381 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-model-cache\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480671 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-tmp-dir\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480870 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-home\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.480955 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.480915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-model-cache\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.482705 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.482678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-dshm\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.483003 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.482987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c90f719-277e-4f8a-b672-046d026cdd3c-tls-certs\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.488642 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.488618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qr4p\" (UniqueName: \"kubernetes.io/projected/5c90f719-277e-4f8a-b672-046d026cdd3c-kube-api-access-2qr4p\") pod \"distinct-w0-b0573c-kserve-748bd8fc59-tqp9p\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.572756 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.572723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.620330 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.620296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:57:50.673647 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.673579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.773087 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.773053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.897926 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.897836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:50.898802 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:50.898781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"]
Sep 12 02:57:50.900767 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:57:50.900735    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c90f719_277e_4f8a_b672_046d026cdd3c.slice/crio-2a00fb3c8661098b1dfff10c66fd39797c5048cc31122432f7856520e740a154 WatchSource:0}: Error finding container 2a00fb3c8661098b1dfff10c66fd39797c5048cc31122432f7856520e740a154: Status 404 returned error can't find the container with id 2a00fb3c8661098b1dfff10c66fd39797c5048cc31122432f7856520e740a154
Sep 12 02:57:51.859656 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:51.859618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p" event={"ID":"5c90f719-277e-4f8a-b672-046d026cdd3c","Type":"ContainerStarted","Data":"6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b"}
Sep 12 02:57:51.859656 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:51.859659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p" event={"ID":"5c90f719-277e-4f8a-b672-046d026cdd3c","Type":"ContainerStarted","Data":"2a00fb3c8661098b1dfff10c66fd39797c5048cc31122432f7856520e740a154"}
Sep 12 02:57:51.877603 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:51.877554    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p" podStartSLOduration=1.8775386539999999 podStartE2EDuration="1.877538654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:57:51.875052833 +0000 UTC m=+1290.157966418" watchObservedRunningTime="2026-09-12 02:57:51.877538654 +0000 UTC m=+1290.160452234"
Sep 12 02:57:53.583341 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:53.583303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:56.161757 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:56.161722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:56.224882 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:56.224842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:57.104233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:57.104205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:57.409523 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:57.409494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:57.671382 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:57.671300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:57:59.453492 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:57:59.453441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:00.086103 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.086072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:00.185507 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.185474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:00.284338 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.284311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:00.620869 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.620830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:58:00.621257 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.620919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:58:00.633899 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.633879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:58:00.844266 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.844236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:00.916747 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:00.916667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 02:58:01.624154 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:01.624121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:02.341346 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:02.341316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:03.125787 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:03.125760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:03.226144 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:03.226115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:03.327044 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:03.327020    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:03.571019 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:03.570947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:03.644219 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:03.644190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:13.049817 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:13.049779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:14.517836 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:14.517808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:24.933679 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:24.933641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:26.233132 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:26.233096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:26.945753 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:26.945721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:27.591890 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:27.591857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:28.344352 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:28.344323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:28.973673 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:28.973637    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:29.749288 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:29.749255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:30.279667 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.279631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:30.635867 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.635837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:30.919801 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.919682    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"]
Sep 12 02:58:30.924340 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.924319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.927273 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.927241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d2effd-openshift-default-dockercfg-zwmdt\""
Sep 12 02:58:30.935105 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.935083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"]
Sep 12 02:58:30.946169 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.946151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:30.972479 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjwl\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-kube-api-access-ssjwl\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972603 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-token\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972603 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istiod-ca-cert\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972692 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-socket\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972692 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-credential-socket\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972692 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-data\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972799 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-podinfo\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-envoy\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:30.972850 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:30.972828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-certs\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074318 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-envoy\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-certs\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjwl\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-kube-api-access-ssjwl\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-token\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istiod-ca-cert\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074525 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-socket\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-credential-socket\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-data\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-podinfo\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074974 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-certs\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.074974 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-socket\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.075089 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.074969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-data\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.075147 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.075138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istiod-ca-cert\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.075201 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.075171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-credential-socket\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.076744 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.076729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-envoy\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.077016 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.076999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-podinfo\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.082233 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.082201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-token\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.082328 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.082317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjwl\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-kube-api-access-ssjwl\") pod \"e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") " pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.236627 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.236548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:31.339140 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.339106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:31.510636 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.510614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"]
Sep 12 02:58:31.521101 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.521050    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:58:31.521191 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.521132    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:58:31.521191 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:31.521169    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:58:32.065503 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:32.065471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" event={"ID":"4802f8c1-a1e6-4768-83f4-016ad641e9a9","Type":"ContainerStarted","Data":"dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"}
Sep 12 02:58:32.065677 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:32.065554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" event={"ID":"4802f8c1-a1e6-4768-83f4-016ad641e9a9","Type":"ContainerStarted","Data":"5bb623959783a7522808fd2051c9810ca0e4fd08ca75cee998840b285b1589e8"}
Sep 12 02:58:32.086584 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:32.086539    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podStartSLOduration=2.086524474 podStartE2EDuration="2.086524474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:58:32.084067158 +0000 UTC m=+1330.366980739" watchObservedRunningTime="2026-09-12 02:58:32.086524474 +0000 UTC m=+1330.369438052"
Sep 12 02:58:32.237797 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:32.237765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:32.245864 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:32.245841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:33.071260 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:33.071227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:33.072290 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:33.072265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:33.330253 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:33.330160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:34.015160 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:34.015120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:34.123585 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:34.123552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:34.247838 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:34.247801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:34.945014 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:34.944988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:35.713063 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:35.713034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:36.411327 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:36.411300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:37.151362 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:37.151329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:38.021551 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.021518    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"]
Sep 12 02:58:38.026502 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.026441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.028866 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.028831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d2effd-serving-cert\""
Sep 12 02:58:38.038350 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.038323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"]
Sep 12 02:58:38.063469 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.061307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:38.120106 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.118716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:38.143050 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.143012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.143252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.143088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.143252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.143123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhwfv\" (UniqueName: \"kubernetes.io/projected/eed29cc7-8ac8-45dc-a650-409b2ec424dc-kube-api-access-fhwfv\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.143252 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.143167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed29cc7-8ac8-45dc-a650-409b2ec424dc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.244328 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.244290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.244787 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.244363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwfv\" (UniqueName: \"kubernetes.io/projected/eed29cc7-8ac8-45dc-a650-409b2ec424dc-kube-api-access-fhwfv\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.244787 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.244427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed29cc7-8ac8-45dc-a650-409b2ec424dc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.244787 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.244580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.244787 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:58:38.244590    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-serving-cert: secret "maas-api-e2e-shared-b-w2-d2effd-serving-cert" not found
Sep 12 02:58:38.244787 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:58:38.244666    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls podName:eed29cc7-8ac8-45dc-a650-409b2ec424dc nodeName:}" failed. No retries permitted until 2026-09-12 02:58:38.744644033 +0000 UTC m=+1337.027557591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" (UID: "eed29cc7-8ac8-45dc-a650-409b2ec424dc") : secret "maas-api-e2e-shared-b-w2-d2effd-serving-cert" not found
Sep 12 02:58:38.245365 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.245332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed29cc7-8ac8-45dc-a650-409b2ec424dc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.247809 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.247777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.253051 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.253028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwfv\" (UniqueName: \"kubernetes.io/projected/eed29cc7-8ac8-45dc-a650-409b2ec424dc-kube-api-access-fhwfv\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.332598 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.332509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:38.520723 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.520641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:38.624661 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.624634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:38.751120 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.751074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.766762 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.766725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:38.941924 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:38.941846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:39.242270 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:39.242241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"]
Sep 12 02:58:39.243531 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:58:39.243503    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed29cc7_8ac8_45dc_a650_409b2ec424dc.slice/crio-cf122f196ed722824a1b51bd4822b56957af09374124f9da83c6065c4f5c5f68 WatchSource:0}: Error finding container cf122f196ed722824a1b51bd4822b56957af09374124f9da83c6065c4f5c5f68: Status 404 returned error can't find the container with id cf122f196ed722824a1b51bd4822b56957af09374124f9da83c6065c4f5c5f68
Sep 12 02:58:39.244928 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:39.244909    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:58:40.115490 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:40.115441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" event={"ID":"eed29cc7-8ac8-45dc-a650-409b2ec424dc","Type":"ContainerStarted","Data":"83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609"}
Sep 12 02:58:40.115662 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:40.115510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" event={"ID":"eed29cc7-8ac8-45dc-a650-409b2ec424dc","Type":"ContainerStarted","Data":"cf122f196ed722824a1b51bd4822b56957af09374124f9da83c6065c4f5c5f68"}
Sep 12 02:58:40.115662 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:40.115611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:40.132198 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:40.132131    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" podStartSLOduration=1.761035685 podStartE2EDuration="2.132117647s" totalImagesPullingTime="371.081962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:58:40.131140913 +0000 UTC m=+1338.414054491" watchObservedRunningTime="2026-09-12 02:58:40.132117647 +0000 UTC m=+1338.415031226"
Sep 12 02:58:46.126481 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:46.126385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 02:58:48.069679 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:48.069633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:48.325806 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:48.325733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:48.426131 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:48.426101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:53.466143 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:53.466088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:58:57.222186 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:57.222153    2579 generic.go:356] "Generic (PLEG): container finished" podID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerID="dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d" exitCode=0
Sep 12 02:58:57.222645 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:57.222214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" event={"ID":"4802f8c1-a1e6-4768-83f4-016ad641e9a9","Type":"ContainerDied","Data":"dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"}
Sep 12 02:58:57.222687 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:57.222644    2579 scope.go:117] "RemoveContainer" containerID="dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"
Sep 12 02:58:57.223788 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:57.223758    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:58:57.223883 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:57.223826    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:58:57.223883 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:57.223855    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 02:58:58.229000 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:58.228966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" event={"ID":"4802f8c1-a1e6-4768-83f4-016ad641e9a9","Type":"ContainerStarted","Data":"37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb"}
Sep 12 02:58:58.237256 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:58.237238    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:58:58.237504 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:58.237483    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 12 02:58:58.237565 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:58.237538    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 12 02:58:59.237275 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:59.237235    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 12 02:58:59.237687 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:58:59.237286    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 12 02:59:00.237403 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:00.237369    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 12 02:59:00.237813 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:00.237429    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 12 02:59:01.131172 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:01.131144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:01.237750 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:01.237716    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 12 02:59:01.238123 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:01.237779    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 12 02:59:01.971615 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:01.971578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:02.237840 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:02.237753    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 12 02:59:02.237840 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:02.237808    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 12 02:59:02.680833 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:02.680802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:03.071564 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:03.071485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:59:03.394149 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:03.394113    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:59:03.396678 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:03.396655    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 02:59:03.545383 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:03.545351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:04.152687 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:04.152659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:04.250366 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:04.250336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:04.878657 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:04.878629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:05.656124 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:05.656095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:09.551795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:09.551763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:10.607104 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:10.607072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:12.984737 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:12.984705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:13.283492 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:13.283397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:13.383532 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:13.383486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:16.039045 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:16.039013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:16.650644 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:16.650596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:16.751185 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:16.751151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:16.855576 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:16.855522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:17.530659 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:17.530616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:18.229679 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:18.229648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:18.963710 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:18.963678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:19.656055 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:19.656021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:20.455656 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:20.455624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:21.156655 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.156628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:21.206913 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.206888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:21.599916 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.599835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:21.687001 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.686972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:21.721897 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.721865    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"]
Sep 12 02:59:21.725521 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.725500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:21.728406 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.728382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1c9060bc-dockercfg-mpf9j\""
Sep 12 02:59:21.728544 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.728387    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1c9060bc\""
Sep 12 02:59:21.735735 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.735713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"]
Sep 12 02:59:21.752988 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.752956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:21.780289 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.780262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") " pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:21.780396 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.780297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxc64\" (UniqueName: \"kubernetes.io/projected/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-kube-api-access-dxc64\") pod \"payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") " pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:21.881694 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.881657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") " pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:21.881694 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.881697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxc64\" (UniqueName: \"kubernetes.io/projected/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-kube-api-access-dxc64\") pod \"payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") " pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:21.882407 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.882384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") " pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:21.890232 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:21.890203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxc64\" (UniqueName: \"kubernetes.io/projected/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-kube-api-access-dxc64\") pod \"payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") " pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:22.036613 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:22.036574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:22.311786 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:22.311749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"]
Sep 12 02:59:22.313104 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:59:22.313078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4b0ea5_c3d7_4e98_9180_90bd55d8f2d3.slice/crio-08f81a93f03f51a872e4d82daa83952dc91616a4ad0751ab4321222133b77c0c WatchSource:0}: Error finding container 08f81a93f03f51a872e4d82daa83952dc91616a4ad0751ab4321222133b77c0c: Status 404 returned error can't find the container with id 08f81a93f03f51a872e4d82daa83952dc91616a4ad0751ab4321222133b77c0c
Sep 12 02:59:22.355265 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:22.355238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg" event={"ID":"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3","Type":"ContainerStarted","Data":"08f81a93f03f51a872e4d82daa83952dc91616a4ad0751ab4321222133b77c0c"}
Sep 12 02:59:22.626829 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:22.626797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:23.256733 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:23.256704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:23.362685 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:23.362639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg" event={"ID":"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3","Type":"ContainerStarted","Data":"5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55"}
Sep 12 02:59:23.362856 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:23.362703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:23.379601 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:23.379555    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg" podStartSLOduration=2.379538614 podStartE2EDuration="2.379538614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:23.377738917 +0000 UTC m=+1381.660652499" watchObservedRunningTime="2026-09-12 02:59:23.379538614 +0000 UTC m=+1381.662452194"
Sep 12 02:59:23.878972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:23.878931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:24.642906 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:24.642872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:25.258404 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:25.258377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:25.877529 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:25.877492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:26.650844 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:26.650810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:27.113994 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:27.113970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:27.272899 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:27.272864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:30.124471 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:30.124416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:30.309266 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:30.309235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.104145 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.104112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.205908 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.205878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.691644 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.691612    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"]
Sep 12 02:59:32.691890 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.691866    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj" podUID="2b378e1a-dc2b-4357-8ebe-e901c475411d" containerName="maas-api" containerID="cri-o://7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595" gracePeriod=30
Sep 12 02:59:32.706871 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.706844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.778796 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.778759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.837625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.837590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.901123 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.901025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:32.936799 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:32.936773    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:59:33.035214 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.035116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:33.098763 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.098723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b378e1a-dc2b-4357-8ebe-e901c475411d-openshift-service-ca\") pod \"2b378e1a-dc2b-4357-8ebe-e901c475411d\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") "
Sep 12 02:59:33.099738 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.099617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b378e1a-dc2b-4357-8ebe-e901c475411d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b378e1a-dc2b-4357-8ebe-e901c475411d" (UID: "2b378e1a-dc2b-4357-8ebe-e901c475411d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:59:33.099893 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.099843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pmxb\" (UniqueName: \"kubernetes.io/projected/2b378e1a-dc2b-4357-8ebe-e901c475411d-kube-api-access-9pmxb\") pod \"2b378e1a-dc2b-4357-8ebe-e901c475411d\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") "
Sep 12 02:59:33.106473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.101617    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-metrics-certs\") pod \"2b378e1a-dc2b-4357-8ebe-e901c475411d\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") "
Sep 12 02:59:33.106473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.101683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-maas-api-tls\") pod \"2b378e1a-dc2b-4357-8ebe-e901c475411d\" (UID: \"2b378e1a-dc2b-4357-8ebe-e901c475411d\") "
Sep 12 02:59:33.106473 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.102041    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b378e1a-dc2b-4357-8ebe-e901c475411d-openshift-service-ca\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:59:33.115206 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.115130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b378e1a-dc2b-4357-8ebe-e901c475411d" (UID: "2b378e1a-dc2b-4357-8ebe-e901c475411d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:59:33.117367 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.117261    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b378e1a-dc2b-4357-8ebe-e901c475411d" (UID: "2b378e1a-dc2b-4357-8ebe-e901c475411d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:59:33.117591 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.117564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b378e1a-dc2b-4357-8ebe-e901c475411d-kube-api-access-9pmxb" (OuterVolumeSpecName: "kube-api-access-9pmxb") pod "2b378e1a-dc2b-4357-8ebe-e901c475411d" (UID: "2b378e1a-dc2b-4357-8ebe-e901c475411d"). InnerVolumeSpecName "kube-api-access-9pmxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:59:33.202914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.202885    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:59:33.202914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.202913    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b378e1a-dc2b-4357-8ebe-e901c475411d-maas-api-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:59:33.202914 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.202922    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmxb\" (UniqueName: \"kubernetes.io/projected/2b378e1a-dc2b-4357-8ebe-e901c475411d-kube-api-access-9pmxb\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 02:59:33.275622 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.275570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:33.415791 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.415759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:33.416893 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.416868    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b378e1a-dc2b-4357-8ebe-e901c475411d" containerID="7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595" exitCode=0
Sep 12 02:59:33.417018 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.416913    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"
Sep 12 02:59:33.417018 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.416949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj" event={"ID":"2b378e1a-dc2b-4357-8ebe-e901c475411d","Type":"ContainerDied","Data":"7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595"}
Sep 12 02:59:33.417018 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.417005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj" event={"ID":"2b378e1a-dc2b-4357-8ebe-e901c475411d","Type":"ContainerDied","Data":"7118fbf16cd65c4cbfe19e3756df58bd97a7b71ed8036df6b8b81dfb983c37b9"}
Sep 12 02:59:33.417121 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.417034    2579 scope.go:117] "RemoveContainer" containerID="7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595"
Sep 12 02:59:33.426637 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.426620    2579 scope.go:117] "RemoveContainer" containerID="7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595"
Sep 12 02:59:33.426907 ip-10-0-128-128 kubenswrapper[2579]: E0912 02:59:33.426887    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595\": container with ID starting with 7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595 not found: ID does not exist" containerID="7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595"
Sep 12 02:59:33.426972 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.426920    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595"} err="failed to get container status \"7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595\": rpc error: code = NotFound desc = could not find container \"7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595\": container with ID starting with 7dfd1a3feef2082539be56c7d0809504b6559c3ccc9571de4d46e79f2c511595 not found: ID does not exist"
Sep 12 02:59:33.440015 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.439991    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"]
Sep 12 02:59:33.443832 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:33.443810    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a07b10-5458b9ff76-dtwdj"]
Sep 12 02:59:34.073258 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:34.073229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:34.245054 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:34.245025    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b378e1a-dc2b-4357-8ebe-e901c475411d" path="/var/lib/kubelet/pods/2b378e1a-dc2b-4357-8ebe-e901c475411d/volumes"
Sep 12 02:59:34.370231 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:34.370206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 02:59:34.711795 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:34.711704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:35.649399 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:35.649342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:36.433290 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:36.433253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:36.556796 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:36.556758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:37.138586 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:37.138543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:37.848329 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:37.848289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:38.304933 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:38.304901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:38.642720 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:38.642682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:39.252106 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:39.252075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:39.875623 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:39.875586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:40.643602 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:40.643575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:41.233304 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:41.233272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:42.422232 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:42.422205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:44.581783 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:44.581753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:44.681912 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:44.681883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:47.952701 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:47.952672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:48.818627 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.818591    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"]
Sep 12 02:59:48.819347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.819309    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b378e1a-dc2b-4357-8ebe-e901c475411d" containerName="maas-api"
Sep 12 02:59:48.819347 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.819335    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b378e1a-dc2b-4357-8ebe-e901c475411d" containerName="maas-api"
Sep 12 02:59:48.819584 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.819425    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b378e1a-dc2b-4357-8ebe-e901c475411d" containerName="maas-api"
Sep 12 02:59:48.824134 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.824114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:48.826915 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.826896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b0573c\"/\"distinct-2-w0-b0573c-kserve-self-signed-certs\""
Sep 12 02:59:48.830860 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.830832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"]
Sep 12 02:59:48.961969 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.961925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jp5\" (UniqueName: \"kubernetes.io/projected/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-kube-api-access-74jp5\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:48.962427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.962032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-home\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:48.962427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.962064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tls-certs\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:48.962427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.962131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-model-cache\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:48.962427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.962165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tmp-dir\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:48.962427 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:48.962185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-dshm\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063222 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-model-cache\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063416 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tmp-dir\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063416 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-dshm\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063416 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jp5\" (UniqueName: \"kubernetes.io/projected/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-kube-api-access-74jp5\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-home\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063625 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tls-certs\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.063881 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.063849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tmp-dir\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.064083 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.064056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-home\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.064200 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.064106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-model-cache\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.066197 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.066169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tls-certs\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.066358 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.066338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-dshm\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.071567 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.071505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jp5\" (UniqueName: \"kubernetes.io/projected/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-kube-api-access-74jp5\") pod \"distinct-2-w0-b0573c-kserve-868759699b-97fkf\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") " pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.136348 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.136234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:49.419681 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.419651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"]
Sep 12 02:59:49.422808 ip-10-0-128-128 kubenswrapper[2579]: W0912 02:59:49.422769    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7e2bb2_01fe_41bb_a3e2_04f25ea53431.slice/crio-7399356ea8c629d5340b71d26e9da9f713645db207bc545001b29ff2b0fc4089 WatchSource:0}: Error finding container 7399356ea8c629d5340b71d26e9da9f713645db207bc545001b29ff2b0fc4089: Status 404 returned error can't find the container with id 7399356ea8c629d5340b71d26e9da9f713645db207bc545001b29ff2b0fc4089
Sep 12 02:59:49.503550 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.503509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf" event={"ID":"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431","Type":"ContainerStarted","Data":"4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b"}
Sep 12 02:59:49.503706 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.503566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf" event={"ID":"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431","Type":"ContainerStarted","Data":"7399356ea8c629d5340b71d26e9da9f713645db207bc545001b29ff2b0fc4089"}
Sep 12 02:59:49.519196 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.519150    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf" podStartSLOduration=1.519133553 podStartE2EDuration="1.519133553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:59:49.51843347 +0000 UTC m=+1407.801347051" watchObservedRunningTime="2026-09-12 02:59:49.519133553 +0000 UTC m=+1407.802047134"
Sep 12 02:59:49.644183 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:49.644151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:54.670192 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:54.670160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:56.428033 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:56.427987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 02:59:59.136898 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:59.136865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:59.137292 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:59.136911    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:59.149178 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:59.149149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:59.566235 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:59.566138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 02:59:59.758956 ip-10-0-128-128 kubenswrapper[2579]: I0912 02:59:59.758922    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:00.141164 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.141131    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"]
Sep 12 03:00:00.144944 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.144928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"
Sep 12 03:00:00.147603 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.147584    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4g6hp\""
Sep 12 03:00:00.153899 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.153873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"]
Sep 12 03:00:00.159005 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.158982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"
Sep 12 03:00:00.430439 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.430408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"]
Sep 12 03:00:00.431432 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:00:00.431408    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c660ca_99e5_4860_8ab0_834633c68b37.slice/crio-b5fa3588a48b07c9395f1423e90aa9214ba33e93ebfee729c87eff68a5b5680d WatchSource:0}: Error finding container b5fa3588a48b07c9395f1423e90aa9214ba33e93ebfee729c87eff68a5b5680d: Status 404 returned error can't find the container with id b5fa3588a48b07c9395f1423e90aa9214ba33e93ebfee729c87eff68a5b5680d
Sep 12 03:00:00.561688 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:00.561656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg" event={"ID":"52c660ca-99e5-4860-8ab0-834633c68b37","Type":"ContainerStarted","Data":"b5fa3588a48b07c9395f1423e90aa9214ba33e93ebfee729c87eff68a5b5680d"}
Sep 12 03:00:03.580246 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:03.580214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg" event={"ID":"52c660ca-99e5-4860-8ab0-834633c68b37","Type":"ContainerStarted","Data":"2326d2d167d02d2111029bfef6411e803e54eb98bd5cbfac4a0c5d26dfd5a727"}
Sep 12 03:00:03.595296 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:03.595248    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg" podStartSLOduration=1.237844019 podStartE2EDuration="3.595234563s" totalImagesPullingTime="2.357390544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:00:03.593431337 +0000 UTC m=+1421.876344915" watchObservedRunningTime="2026-09-12 03:00:03.595234563 +0000 UTC m=+1421.878148143"
Sep 12 03:00:04.586913 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:04.586875    2579 generic.go:356] "Generic (PLEG): container finished" podID="52c660ca-99e5-4860-8ab0-834633c68b37" containerID="2326d2d167d02d2111029bfef6411e803e54eb98bd5cbfac4a0c5d26dfd5a727" exitCode=0
Sep 12 03:00:04.587342 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:04.586932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg" event={"ID":"52c660ca-99e5-4860-8ab0-834633c68b37","Type":"ContainerDied","Data":"2326d2d167d02d2111029bfef6411e803e54eb98bd5cbfac4a0c5d26dfd5a727"}
Sep 12 03:00:05.795436 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:05.795412    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"
Sep 12 03:00:06.600897 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:06.600798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg" event={"ID":"52c660ca-99e5-4860-8ab0-834633c68b37","Type":"ContainerDied","Data":"b5fa3588a48b07c9395f1423e90aa9214ba33e93ebfee729c87eff68a5b5680d"}
Sep 12 03:00:06.600897 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:06.600823    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819700-n2sjg"
Sep 12 03:00:06.600897 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:06.600855    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5fa3588a48b07c9395f1423e90aa9214ba33e93ebfee729c87eff68a5b5680d"
Sep 12 03:00:14.896403 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:14.896324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:15.163005 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:15.162921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:15.718767 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:15.718734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:16.443687 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:16.443655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:17.045971 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:17.045944    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:17.213990 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:17.213958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:17.851613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:17.851582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:18.545530 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:18.545491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:19.219676 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:19.219644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:19.849781 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:19.849747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:20.556709 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:20.556678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:25.953663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:25.953630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:25.977869 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:25.977835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"]
Sep 12 03:00:25.978156 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:25.978096    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf" podUID="41dc4e4e-28c8-486d-b99d-d765e7f28a98" containerName="payload-processing" containerID="cri-o://cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547" gracePeriod=30
Sep 12 03:00:25.987426 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:25.987405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"]
Sep 12 03:00:25.987675 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:25.987617    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh" podUID="9e414f93-adf3-41a5-b49f-a4ec1befe24e" containerName="payload-pre-processing" containerID="cri-o://322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f" gracePeriod=30
Sep 12 03:00:26.018700 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.018676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:26.118219 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.118157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:26.173863 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.173834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:26.281666 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.281642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 03:00:26.284629 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.284604    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 03:00:26.314979 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.313781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhpcs\" (UniqueName: \"kubernetes.io/projected/9e414f93-adf3-41a5-b49f-a4ec1befe24e-kube-api-access-jhpcs\") pod \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") "
Sep 12 03:00:26.314979 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.313877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7rln\" (UniqueName: \"kubernetes.io/projected/41dc4e4e-28c8-486d-b99d-d765e7f28a98-kube-api-access-n7rln\") pod \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") "
Sep 12 03:00:26.314979 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.313968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dc4e4e-28c8-486d-b99d-d765e7f28a98-plugins-config-volume\") pod \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\" (UID: \"41dc4e4e-28c8-486d-b99d-d765e7f28a98\") "
Sep 12 03:00:26.314979 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.314024    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e414f93-adf3-41a5-b49f-a4ec1befe24e-plugins-config-volume\") pod \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\" (UID: \"9e414f93-adf3-41a5-b49f-a4ec1befe24e\") "
Sep 12 03:00:26.316069 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.315728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e414f93-adf3-41a5-b49f-a4ec1befe24e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e414f93-adf3-41a5-b49f-a4ec1befe24e" (UID: "9e414f93-adf3-41a5-b49f-a4ec1befe24e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:00:26.318274 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.317597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dc4e4e-28c8-486d-b99d-d765e7f28a98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41dc4e4e-28c8-486d-b99d-d765e7f28a98" (UID: "41dc4e4e-28c8-486d-b99d-d765e7f28a98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:00:26.319808 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.319773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e414f93-adf3-41a5-b49f-a4ec1befe24e-kube-api-access-jhpcs" (OuterVolumeSpecName: "kube-api-access-jhpcs") pod "9e414f93-adf3-41a5-b49f-a4ec1befe24e" (UID: "9e414f93-adf3-41a5-b49f-a4ec1befe24e"). InnerVolumeSpecName "kube-api-access-jhpcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:00:26.322040 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.322001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dc4e4e-28c8-486d-b99d-d765e7f28a98-kube-api-access-n7rln" (OuterVolumeSpecName: "kube-api-access-n7rln") pod "41dc4e4e-28c8-486d-b99d-d765e7f28a98" (UID: "41dc4e4e-28c8-486d-b99d-d765e7f28a98"). InnerVolumeSpecName "kube-api-access-n7rln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:00:26.416826 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.416792    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dc4e4e-28c8-486d-b99d-d765e7f28a98-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.416826 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.416821    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e414f93-adf3-41a5-b49f-a4ec1befe24e-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.416826 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.416832    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpcs\" (UniqueName: \"kubernetes.io/projected/9e414f93-adf3-41a5-b49f-a4ec1befe24e-kube-api-access-jhpcs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.417057 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.416840    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7rln\" (UniqueName: \"kubernetes.io/projected/41dc4e4e-28c8-486d-b99d-d765e7f28a98-kube-api-access-n7rln\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:26.515765 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.515684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:26.617327 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.617290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:26.712099 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.712061    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e414f93-adf3-41a5-b49f-a4ec1befe24e" containerID="322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f" exitCode=0
Sep 12 03:00:26.712099 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.712103    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"
Sep 12 03:00:26.712360 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.712147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh" event={"ID":"9e414f93-adf3-41a5-b49f-a4ec1befe24e","Type":"ContainerDied","Data":"322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f"}
Sep 12 03:00:26.712360 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.712200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh" event={"ID":"9e414f93-adf3-41a5-b49f-a4ec1befe24e","Type":"ContainerDied","Data":"ac2772fb4e14013e09becf0e766f651c1f0cd7ed1b0cc9f4072ff3be46246d67"}
Sep 12 03:00:26.712360 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.712231    2579 scope.go:117] "RemoveContainer" containerID="322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f"
Sep 12 03:00:26.713698 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.713679    2579 generic.go:356] "Generic (PLEG): container finished" podID="41dc4e4e-28c8-486d-b99d-d765e7f28a98" containerID="cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547" exitCode=0
Sep 12 03:00:26.713814 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.713737    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"
Sep 12 03:00:26.713814 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.713747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf" event={"ID":"41dc4e4e-28c8-486d-b99d-d765e7f28a98","Type":"ContainerDied","Data":"cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547"}
Sep 12 03:00:26.713814 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.713789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf" event={"ID":"41dc4e4e-28c8-486d-b99d-d765e7f28a98","Type":"ContainerDied","Data":"ff20ec00fe349a92fbfbbe874623a027743e7f26c2b2054ace25e630874dc9f2"}
Sep 12 03:00:26.722766 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.722747    2579 scope.go:117] "RemoveContainer" containerID="322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f"
Sep 12 03:00:26.723074 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:00:26.723048    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f\": container with ID starting with 322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f not found: ID does not exist" containerID="322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f"
Sep 12 03:00:26.723147 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.723084    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f"} err="failed to get container status \"322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f\": rpc error: code = NotFound desc = could not find container \"322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f\": container with ID starting with 322014f51178bd3edf86547e16159c058ecb30d8ee451ac13d6ee71d8135e27f not found: ID does not exist"
Sep 12 03:00:26.723147 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.723100    2579 scope.go:117] "RemoveContainer" containerID="cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547"
Sep 12 03:00:26.731324 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.731307    2579 scope.go:117] "RemoveContainer" containerID="cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547"
Sep 12 03:00:26.731636 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:00:26.731616    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547\": container with ID starting with cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547 not found: ID does not exist" containerID="cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547"
Sep 12 03:00:26.731697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.731644    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547"} err="failed to get container status \"cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547\": rpc error: code = NotFound desc = could not find container \"cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547\": container with ID starting with cd1a140985ab12484a3ba8a5b3e37a571d3f04957398ecd4ed90dc39875be547 not found: ID does not exist"
Sep 12 03:00:26.747275 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.747255    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"]
Sep 12 03:00:26.749320 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.749301    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f93121-6c544b8b55-2c6hf"]
Sep 12 03:00:26.764786 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.764764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"]
Sep 12 03:00:26.771541 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:26.771483    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f93121-dbb8945fd-lcfrh"]
Sep 12 03:00:27.359937 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:27.359908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:27.462884 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:27.462833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:27.560736 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:27.560665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:28.126211 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:28.126183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:28.246902 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:28.246867    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41dc4e4e-28c8-486d-b99d-d765e7f28a98" path="/var/lib/kubelet/pods/41dc4e4e-28c8-486d-b99d-d765e7f28a98/volumes"
Sep 12 03:00:28.247369 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:28.247352    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e414f93-adf3-41a5-b49f-a4ec1befe24e" path="/var/lib/kubelet/pods/9e414f93-adf3-41a5-b49f-a4ec1befe24e/volumes"
Sep 12 03:00:28.875795 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:28.875739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:29.552629 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:29.552593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:30.176656 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:30.176620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:32.314055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:32.313996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:32.368232 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:32.368204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:32.575034 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:32.574944    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"]
Sep 12 03:00:32.575189 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:32.575164    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy" containerID="cri-o://95d287196d4d656d6e237bbaecc30661f7e972fd920e12686f3ba5c8709c4771" gracePeriod=30
Sep 12 03:00:32.594053 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:32.594024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:33.203492 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:33.203431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:33.596191 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:33.596118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:33.695137 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:33.695109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:35.637564 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:35.637534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:36.249427 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:36.249392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:36.348768 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:36.348731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:36.404235 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:36.404205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:37.236285 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.236243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:37.776252 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.776219    2579 generic.go:356] "Generic (PLEG): container finished" podID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerID="95d287196d4d656d6e237bbaecc30661f7e972fd920e12686f3ba5c8709c4771" exitCode=0
Sep 12 03:00:37.776401 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.776284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" event={"ID":"b57f3ab4-773a-4f5b-a706-bb0e272868b3","Type":"ContainerDied","Data":"95d287196d4d656d6e237bbaecc30661f7e972fd920e12686f3ba5c8709c4771"}
Sep 12 03:00:37.776401 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.776342    2579 scope.go:117] "RemoveContainer" containerID="077dceae8828c79bdaaf46d5536e900aa0fdf313f4a940d81754c4dff448db6d"
Sep 12 03:00:37.821321 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.821300    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 03:00:37.926442 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86n5j\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-kube-api-access-86n5j\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926442 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-certs\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-token\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istiod-ca-cert\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-credential-socket\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926618    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-socket\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-envoy\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.926727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-podinfo\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.927043 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-data\") pod \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\" (UID: \"b57f3ab4-773a-4f5b-a706-bb0e272868b3\") "
Sep 12 03:00:37.927043 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926777    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:00:37.927043 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:00:37.927043 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:00:37.927043 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.926989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:00:37.927242 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.927112    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:37.927242 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.927132    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:37.927242 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.927142    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:37.927242 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.927151    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:37.927242 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.927114    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:00:37.928970 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.928937    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-kube-api-access-86n5j" (OuterVolumeSpecName: "kube-api-access-86n5j") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "kube-api-access-86n5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:00:37.928970 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.928932    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:00:37.929288 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.929269    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:00:37.929346 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:37.929271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b57f3ab4-773a-4f5b-a706-bb0e272868b3" (UID: "b57f3ab4-773a-4f5b-a706-bb0e272868b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:00:38.028228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.028193    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:38.028228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.028222    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:38.028228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.028230    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:38.028228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.028239    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86n5j\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-kube-api-access-86n5j\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:38.028512 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.028248    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57f3ab4-773a-4f5b-a706-bb0e272868b3-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:00:38.136868 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.136829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:38.782911 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.782885    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"
Sep 12 03:00:38.783365 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.782911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7" event={"ID":"b57f3ab4-773a-4f5b-a706-bb0e272868b3","Type":"ContainerDied","Data":"fc320679c64a0ac59bac32c640a67ea61de499129f0c03c086ccce61aa877a96"}
Sep 12 03:00:38.783365 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.782965    2579 scope.go:117] "RemoveContainer" containerID="95d287196d4d656d6e237bbaecc30661f7e972fd920e12686f3ba5c8709c4771"
Sep 12 03:00:38.801372 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.801349    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"]
Sep 12 03:00:38.804883 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.804861    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f93121-openshift-default-5f9496f9b8-n2nt7"]
Sep 12 03:00:38.823622 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:38.823602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:40.245602 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.245558    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" path="/var/lib/kubelet/pods/b57f3ab4-773a-4f5b-a706-bb0e272868b3/volumes"
Sep 12 03:00:40.656660 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.656627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:40.721407 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"]
Sep 12 03:00:40.721789 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721776    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c660ca-99e5-4860-8ab0-834633c68b37" containerName="cleanup"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721791    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c660ca-99e5-4860-8ab0-834633c68b37" containerName="cleanup"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721798    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e414f93-adf3-41a5-b49f-a4ec1befe24e" containerName="payload-pre-processing"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721803    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e414f93-adf3-41a5-b49f-a4ec1befe24e" containerName="payload-pre-processing"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721817    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41dc4e4e-28c8-486d-b99d-d765e7f28a98" containerName="payload-processing"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721823    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41dc4e4e-28c8-486d-b99d-d765e7f28a98" containerName="payload-processing"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721843    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy"
Sep 12 03:00:40.721845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721848    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721853    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721858    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721913    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c660ca-99e5-4860-8ab0-834633c68b37" containerName="cleanup"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721924    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41dc4e4e-28c8-486d-b99d-d765e7f28a98" containerName="payload-processing"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721930    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e414f93-adf3-41a5-b49f-a4ec1befe24e" containerName="payload-pre-processing"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721936    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy"
Sep 12 03:00:40.722070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.721941    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57f3ab4-773a-4f5b-a706-bb0e272868b3" containerName="istio-proxy"
Sep 12 03:00:40.726577 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.726558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:40.729181 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.729160    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f91169\""
Sep 12 03:00:40.729345 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.729225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f91169-dockercfg-dtrcx\""
Sep 12 03:00:40.733512 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.733490    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"]
Sep 12 03:00:40.858086 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.858054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsmd\" (UniqueName: \"kubernetes.io/projected/447f380b-3ba0-4a91-a5a3-d987dad090bc-kube-api-access-cnsmd\") pod \"payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:40.858264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.858114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/447f380b-3ba0-4a91-a5a3-d987dad090bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:40.959333 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.959235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/447f380b-3ba0-4a91-a5a3-d987dad090bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:40.959534 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.959360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsmd\" (UniqueName: \"kubernetes.io/projected/447f380b-3ba0-4a91-a5a3-d987dad090bc-kube-api-access-cnsmd\") pod \"payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:40.959955 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.959933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/447f380b-3ba0-4a91-a5a3-d987dad090bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:40.978911 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:40.978883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsmd\" (UniqueName: \"kubernetes.io/projected/447f380b-3ba0-4a91-a5a3-d987dad090bc-kube-api-access-cnsmd\") pod \"payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:41.038284 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:41.038250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:41.311291 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:41.311257    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"]
Sep 12 03:00:41.312818 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:00:41.312790    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447f380b_3ba0_4a91_a5a3_d987dad090bc.slice/crio-e286d8d7326843693b856720aaed36151b232ed49c214391abb213e5e5278f4a WatchSource:0}: Error finding container e286d8d7326843693b856720aaed36151b232ed49c214391abb213e5e5278f4a: Status 404 returned error can't find the container with id e286d8d7326843693b856720aaed36151b232ed49c214391abb213e5e5278f4a
Sep 12 03:00:41.803660 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:41.803624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj" event={"ID":"447f380b-3ba0-4a91-a5a3-d987dad090bc","Type":"ContainerStarted","Data":"7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40"}
Sep 12 03:00:41.803851 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:41.803674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj" event={"ID":"447f380b-3ba0-4a91-a5a3-d987dad090bc","Type":"ContainerStarted","Data":"e286d8d7326843693b856720aaed36151b232ed49c214391abb213e5e5278f4a"}
Sep 12 03:00:41.803851 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:41.803841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:41.819014 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:41.818966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj" podStartSLOduration=1.818948432 podStartE2EDuration="1.818948432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:00:41.817957199 +0000 UTC m=+1460.100870781" watchObservedRunningTime="2026-09-12 03:00:41.818948432 +0000 UTC m=+1460.101862013"
Sep 12 03:00:47.397208 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.397178    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"]
Sep 12 03:00:47.401036 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.401018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.404920 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.404888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"e2e-conflict-b-40f128f2-kserve-self-signed-certs\""
Sep 12 03:00:47.404920 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.404909    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"kube-root-ca.crt\""
Sep 12 03:00:47.404920 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.404916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"default-dockercfg-hl88b\""
Sep 12 03:00:47.405167 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.404922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"openshift-service-ca.crt\""
Sep 12 03:00:47.411600 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.411583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"]
Sep 12 03:00:47.519955 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.519923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-home\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.519955 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.519959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ac3305-ba52-4517-906d-22b7d780da88-tls-certs\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.520213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.520002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-model-cache\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.520213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.520110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltg6j\" (UniqueName: \"kubernetes.io/projected/e4ac3305-ba52-4517-906d-22b7d780da88-kube-api-access-ltg6j\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.520213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.520148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-tmp-dir\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.520213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.520176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-dshm\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621560 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-model-cache\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621757 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg6j\" (UniqueName: \"kubernetes.io/projected/e4ac3305-ba52-4517-906d-22b7d780da88-kube-api-access-ltg6j\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621757 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-tmp-dir\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621757 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-dshm\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621757 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-home\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621993 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ac3305-ba52-4517-906d-22b7d780da88-tls-certs\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.621993 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.621969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-model-cache\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.622074 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.622023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-tmp-dir\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.622110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.622086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-home\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.623965 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.623935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-dshm\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.624264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.624249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ac3305-ba52-4517-906d-22b7d780da88-tls-certs\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.630558 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.630531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltg6j\" (UniqueName: \"kubernetes.io/projected/e4ac3305-ba52-4517-906d-22b7d780da88-kube-api-access-ltg6j\") pod \"e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") " pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.710933 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.710852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:47.809520 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.809480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:00:47.993635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:47.993579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"]
Sep 12 03:00:47.995328 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:00:47.995288    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4ac3305_ba52_4517_906d_22b7d780da88.slice/crio-3b3e11bb2740433089bd91368bccf995fb5ece06ed60a6869a440966cc90c2c2 WatchSource:0}: Error finding container 3b3e11bb2740433089bd91368bccf995fb5ece06ed60a6869a440966cc90c2c2: Status 404 returned error can't find the container with id 3b3e11bb2740433089bd91368bccf995fb5ece06ed60a6869a440966cc90c2c2
Sep 12 03:00:48.840206 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:48.840172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz" event={"ID":"e4ac3305-ba52-4517-906d-22b7d780da88","Type":"ContainerStarted","Data":"cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77"}
Sep 12 03:00:48.840605 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:48.840216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz" event={"ID":"e4ac3305-ba52-4517-906d-22b7d780da88","Type":"ContainerStarted","Data":"3b3e11bb2740433089bd91368bccf995fb5ece06ed60a6869a440966cc90c2c2"}
Sep 12 03:00:48.855743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:48.855696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz" podStartSLOduration=1.855677618 podStartE2EDuration="1.855677618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:00:48.855009525 +0000 UTC m=+1467.137923103" watchObservedRunningTime="2026-09-12 03:00:48.855677618 +0000 UTC m=+1467.138591199"
Sep 12 03:00:52.811942 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:52.811914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:00:57.711440 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:57.711388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:57.711440 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:57.711469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:57.724207 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:57.724181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:00:57.898233 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:00:57.898209    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:01:03.139766 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:03.139733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:03.780962 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:03.780927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:04.444207 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:04.444178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:05.163895 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:05.163855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:05.841388 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:05.841353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:06.403305 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:06.403274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:06.502935 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:06.502901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:06.773040 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:06.772925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:07.078164 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:07.078071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:07.220590 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:07.220553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:07.327600 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:07.327561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:09.884858 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:09.884821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:10.449200 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:10.449162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:10.553355 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:10.553324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:10.652949 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:10.652921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:11.361869 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:11.361831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:12.091067 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:12.091039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:12.757339 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:12.757307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:13.639370 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:13.639342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:14.269317 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:14.269286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:14.896580 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:14.896547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:14.944907 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:14.944872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:15.046485 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.046435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:15.337890 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.337795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:15.436837 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.436795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:15.466426 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.466396    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"]
Sep 12 03:01:15.469967 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.469952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.472697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.472666    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e401bd\""
Sep 12 03:01:15.472697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.472678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e401bd-dockercfg-dq7kd\""
Sep 12 03:01:15.479472 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.478967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"]
Sep 12 03:01:15.579307 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.579274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/e82c544c-60fc-471d-8b66-51a94400b15f-kube-api-access-qgpv6\") pod \"payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.579659 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.579387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82c544c-60fc-471d-8b66-51a94400b15f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.680754 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.680696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82c544c-60fc-471d-8b66-51a94400b15f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.680976 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.680789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/e82c544c-60fc-471d-8b66-51a94400b15f-kube-api-access-qgpv6\") pod \"payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.681540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.681512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82c544c-60fc-471d-8b66-51a94400b15f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.689034 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.689013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/e82c544c-60fc-471d-8b66-51a94400b15f-kube-api-access-qgpv6\") pod \"payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:15.783830 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:15.783793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:16.055550 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:16.055518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"]
Sep 12 03:01:16.057527 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:01:16.057496    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82c544c_60fc_471d_8b66_51a94400b15f.slice/crio-c4ce785618022f24e7f2355abc6da3a9fa3d4dd60a3ac0c9f0b7efe23d973977 WatchSource:0}: Error finding container c4ce785618022f24e7f2355abc6da3a9fa3d4dd60a3ac0c9f0b7efe23d973977: Status 404 returned error can't find the container with id c4ce785618022f24e7f2355abc6da3a9fa3d4dd60a3ac0c9f0b7efe23d973977
Sep 12 03:01:16.989468 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:16.989418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" event={"ID":"e82c544c-60fc-471d-8b66-51a94400b15f","Type":"ContainerStarted","Data":"b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc"}
Sep 12 03:01:16.989863 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:16.989497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" event={"ID":"e82c544c-60fc-471d-8b66-51a94400b15f","Type":"ContainerStarted","Data":"c4ce785618022f24e7f2355abc6da3a9fa3d4dd60a3ac0c9f0b7efe23d973977"}
Sep 12 03:01:16.989863 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:16.989526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:17.005302 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:17.005258    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" podStartSLOduration=2.005242526 podStartE2EDuration="2.005242526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:01:17.004377123 +0000 UTC m=+1495.287290708" watchObservedRunningTime="2026-09-12 03:01:17.005242526 +0000 UTC m=+1495.288156125"
Sep 12 03:01:19.028134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:19.028097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:19.750536 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:19.750503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:20.740228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:20.740190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:21.362521 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:21.362486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:22.031086 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:22.031061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:22.243140 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:22.243113    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:01:22.245200 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:22.245176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:01:22.847265 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:22.847229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:23.519350 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:23.519318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:24.147141 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:24.147107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:24.948643 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:24.948610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:25.299798 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:25.299718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:25.647508 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:25.647479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:26.550140 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:26.550115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:27.180596 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:27.180566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:27.822904 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:27.822868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:27.997839 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:27.997813    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:01:28.654514 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:28.654486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:29.256979 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:29.256947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:30.026536 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:30.026501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:33.140341 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:33.140313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:35.583191 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.583159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:35.615837 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.615743    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"]
Sep 12 03:01:35.616019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.615970    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg" podUID="4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" containerName="payload-processing" containerID="cri-o://5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55" gracePeriod=30
Sep 12 03:01:35.683364 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.683334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:35.774922 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.774885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:35.826899 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.826866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:35.892427 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.892396    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 03:01:35.981421 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.981384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxc64\" (UniqueName: \"kubernetes.io/projected/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-kube-api-access-dxc64\") pod \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") "
Sep 12 03:01:35.981606 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.981545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-plugins-config-volume\") pod \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\" (UID: \"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3\") "
Sep 12 03:01:35.981945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.981918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" (UID: "4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:01:35.983751 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:35.983715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-kube-api-access-dxc64" (OuterVolumeSpecName: "kube-api-access-dxc64") pod "4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" (UID: "4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3"). InnerVolumeSpecName "kube-api-access-dxc64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:01:36.082368 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.082333    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:36.082368 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.082364    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxc64\" (UniqueName: \"kubernetes.io/projected/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3-kube-api-access-dxc64\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:36.090900 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.090874    2579 generic.go:356] "Generic (PLEG): container finished" podID="4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" containerID="5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55" exitCode=0
Sep 12 03:01:36.091035 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.090922    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"
Sep 12 03:01:36.091035 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.090969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg" event={"ID":"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3","Type":"ContainerDied","Data":"5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55"}
Sep 12 03:01:36.091035 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.091018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg" event={"ID":"4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3","Type":"ContainerDied","Data":"08f81a93f03f51a872e4d82daa83952dc91616a4ad0751ab4321222133b77c0c"}
Sep 12 03:01:36.091139 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.091040    2579 scope.go:117] "RemoveContainer" containerID="5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55"
Sep 12 03:01:36.100656 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.100640    2579 scope.go:117] "RemoveContainer" containerID="5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55"
Sep 12 03:01:36.100930 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:01:36.100913    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55\": container with ID starting with 5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55 not found: ID does not exist" containerID="5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55"
Sep 12 03:01:36.100998 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.100952    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55"} err="failed to get container status \"5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55\": rpc error: code = NotFound desc = could not find container \"5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55\": container with ID starting with 5dd96b660f0e40952f05c40136c6bf230a5a6f8996f0189007d75027ebef6d55 not found: ID does not exist"
Sep 12 03:01:36.119796 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.117135    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"]
Sep 12 03:01:36.122698 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.122667    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c9060bc-599c8d8569-vd4kg"]
Sep 12 03:01:36.246074 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.246043    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" path="/var/lib/kubelet/pods/4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3/volumes"
Sep 12 03:01:36.298011 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.297981    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"]
Sep 12 03:01:36.318348 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.318324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:36.699428 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:36.699394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:37.656806 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:37.656776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:38.338786 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:38.338755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:38.963159 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:38.963125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:39.064099 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:39.064071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:39.743724 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:39.743691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:40.355925 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:40.355892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:44.130199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:44.130089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:44.568256 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:44.568169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:47.720047 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.720008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"]
Sep 12 03:01:47.720622 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.720603    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" containerName="payload-processing"
Sep 12 03:01:47.720716 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.720625    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" containerName="payload-processing"
Sep 12 03:01:47.720776 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.720743    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4b0ea5-c3d7-4e98-9180-90bd55d8f2d3" containerName="payload-processing"
Sep 12 03:01:47.724961 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.724940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.727642 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.727624    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"distinct-w4-c6190b-kserve-self-signed-certs\""
Sep 12 03:01:47.731217 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.731197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"]
Sep 12 03:01:47.793313 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.793275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tmp-dir\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.793313 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.793311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-home\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.793567 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.793402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-model-cache\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.793567 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.793474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wshmf\" (UniqueName: \"kubernetes.io/projected/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-kube-api-access-wshmf\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.793567 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.793555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-dshm\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.793740 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.793582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tls-certs\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.823477 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.823419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:01:47.894055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tmp-dir\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-home\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894277 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-model-cache\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894277 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wshmf\" (UniqueName: \"kubernetes.io/projected/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-kube-api-access-wshmf\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894277 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-dshm\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894277 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tls-certs\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894537 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tmp-dir\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894537 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-home\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.894840 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.894816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-model-cache\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.896702 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.896679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-dshm\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.896877 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.896858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tls-certs\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:47.902666 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:47.902645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wshmf\" (UniqueName: \"kubernetes.io/projected/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-kube-api-access-wshmf\") pod \"distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:48.035889 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:48.035790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:48.311568 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:48.311487    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"]
Sep 12 03:01:48.315131 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:01:48.314482    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0765346d_1ae2_488a_a0c6_dc5288ffd9bb.slice/crio-a4b270fc6f16ff6ac783c60df6d874c257890cd9c11abe3b09d9a4aa00820240 WatchSource:0}: Error finding container a4b270fc6f16ff6ac783c60df6d874c257890cd9c11abe3b09d9a4aa00820240: Status 404 returned error can't find the container with id a4b270fc6f16ff6ac783c60df6d874c257890cd9c11abe3b09d9a4aa00820240
Sep 12 03:01:49.163909 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:49.163875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l" event={"ID":"0765346d-1ae2-488a-a0c6-dc5288ffd9bb","Type":"ContainerStarted","Data":"6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa"}
Sep 12 03:01:49.164283 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:49.163918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l" event={"ID":"0765346d-1ae2-488a-a0c6-dc5288ffd9bb","Type":"ContainerStarted","Data":"a4b270fc6f16ff6ac783c60df6d874c257890cd9c11abe3b09d9a4aa00820240"}
Sep 12 03:01:49.179424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:49.179382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l" podStartSLOduration=2.179368433 podStartE2EDuration="2.179368433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:01:49.178364384 +0000 UTC m=+1527.461277967" watchObservedRunningTime="2026-09-12 03:01:49.179368433 +0000 UTC m=+1527.462282013"
Sep 12 03:01:51.319104 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.319046    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz" podUID="e4ac3305-ba52-4517-906d-22b7d780da88" containerName="main" containerID="cri-o://cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77" gracePeriod=45
Sep 12 03:01:51.570469 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.570396    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.628753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-dshm\") pod \"e4ac3305-ba52-4517-906d-22b7d780da88\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") "
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.628880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-home\") pod \"e4ac3305-ba52-4517-906d-22b7d780da88\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") "
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.628933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltg6j\" (UniqueName: \"kubernetes.io/projected/e4ac3305-ba52-4517-906d-22b7d780da88-kube-api-access-ltg6j\") pod \"e4ac3305-ba52-4517-906d-22b7d780da88\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") "
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.628981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-model-cache\") pod \"e4ac3305-ba52-4517-906d-22b7d780da88\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") "
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.629011    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ac3305-ba52-4517-906d-22b7d780da88-tls-certs\") pod \"e4ac3305-ba52-4517-906d-22b7d780da88\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") "
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.629035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-tmp-dir\") pod \"e4ac3305-ba52-4517-906d-22b7d780da88\" (UID: \"e4ac3305-ba52-4517-906d-22b7d780da88\") "
Sep 12 03:01:51.629264 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.629220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4ac3305-ba52-4517-906d-22b7d780da88" (UID: "e4ac3305-ba52-4517-906d-22b7d780da88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:51.629825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.629278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-home" (OuterVolumeSpecName: "home") pod "e4ac3305-ba52-4517-906d-22b7d780da88" (UID: "e4ac3305-ba52-4517-906d-22b7d780da88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:51.629825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.629370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4ac3305-ba52-4517-906d-22b7d780da88" (UID: "e4ac3305-ba52-4517-906d-22b7d780da88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:51.630061 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.630035    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:51.630164 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.630087    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:51.630164 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.630115    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:51.631882 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.631838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-dshm" (OuterVolumeSpecName: "dshm") pod "e4ac3305-ba52-4517-906d-22b7d780da88" (UID: "e4ac3305-ba52-4517-906d-22b7d780da88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:01:51.632017 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.631916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ac3305-ba52-4517-906d-22b7d780da88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4ac3305-ba52-4517-906d-22b7d780da88" (UID: "e4ac3305-ba52-4517-906d-22b7d780da88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:01:51.632083 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.632029    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ac3305-ba52-4517-906d-22b7d780da88-kube-api-access-ltg6j" (OuterVolumeSpecName: "kube-api-access-ltg6j") pod "e4ac3305-ba52-4517-906d-22b7d780da88" (UID: "e4ac3305-ba52-4517-906d-22b7d780da88"). InnerVolumeSpecName "kube-api-access-ltg6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:01:51.730787 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.730758    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltg6j\" (UniqueName: \"kubernetes.io/projected/e4ac3305-ba52-4517-906d-22b7d780da88-kube-api-access-ltg6j\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:51.730787 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.730786    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ac3305-ba52-4517-906d-22b7d780da88-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:51.730787 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:51.730795    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ac3305-ba52-4517-906d-22b7d780da88-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:01:52.188473 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.188417    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4ac3305-ba52-4517-906d-22b7d780da88" containerID="cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77" exitCode=0
Sep 12 03:01:52.188473 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.188480    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"
Sep 12 03:01:52.188708 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.188500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz" event={"ID":"e4ac3305-ba52-4517-906d-22b7d780da88","Type":"ContainerDied","Data":"cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77"}
Sep 12 03:01:52.188708 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.188549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz" event={"ID":"e4ac3305-ba52-4517-906d-22b7d780da88","Type":"ContainerDied","Data":"3b3e11bb2740433089bd91368bccf995fb5ece06ed60a6869a440966cc90c2c2"}
Sep 12 03:01:52.188708 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.188571    2579 scope.go:117] "RemoveContainer" containerID="cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77"
Sep 12 03:01:52.198166 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.198148    2579 scope.go:117] "RemoveContainer" containerID="cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77"
Sep 12 03:01:52.198408 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:01:52.198390    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77\": container with ID starting with cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77 not found: ID does not exist" containerID="cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77"
Sep 12 03:01:52.198471 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.198416    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77"} err="failed to get container status \"cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77\": rpc error: code = NotFound desc = could not find container \"cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77\": container with ID starting with cf1d858bb25d0573cac606d01333e7f018ad2ace91a679f7a5b07d654f779a77 not found: ID does not exist"
Sep 12 03:01:52.209916 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.209897    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"]
Sep 12 03:01:52.213497 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.213476    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/e2e-conflict-b-40f128f2-kserve-6c989c4c6b-dlcpz"]
Sep 12 03:01:52.247723 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:52.247699    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ac3305-ba52-4517-906d-22b7d780da88" path="/var/lib/kubelet/pods/e4ac3305-ba52-4517-906d-22b7d780da88/volumes"
Sep 12 03:01:58.036194 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:58.036160    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:58.036741 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:58.036218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:58.048901 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:58.048880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:01:58.233971 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:01:58.233940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:02:17.922126 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:17.922096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:21.095322 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:21.095287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:21.252731 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:21.252705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:26.737638 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:26.737601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:32.609889 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:32.609859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:32.671136 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:32.671104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:33.662904 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:33.662865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:34.192809 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:34.192782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:34.293800 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:34.293772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:34.393640 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:34.393607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:34.994895 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:34.994862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:35.091525 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:35.091492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:35.234496 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:35.234441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:36.495843 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:36.495808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:42.081048 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:42.081019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:42.181229 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:42.181201    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:42.772285 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:42.772254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:42.873657 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:42.873617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:47.987914 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:47.987881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:48.044059 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:48.044029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:48.562406 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:48.562369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:48.621534 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:48.621498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:48.703059 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:48.703027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:48.807119 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:48.807094    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:48.910055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:48.910022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:51.444743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:51.444709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:52.266206 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:52.266172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:59.306050 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:59.306000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:02:59.550632 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:02:59.550598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:02.217638 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.217604    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"]
Sep 12 03:03:02.218206 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.218186    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ac3305-ba52-4517-906d-22b7d780da88" containerName="main"
Sep 12 03:03:02.218308 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.218209    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ac3305-ba52-4517-906d-22b7d780da88" containerName="main"
Sep 12 03:03:02.218373 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.218308    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ac3305-ba52-4517-906d-22b7d780da88" containerName="main"
Sep 12 03:03:02.221527 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.221505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.223962 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.223944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c6190b\"/\"distinct-2-w4-c6190b-kserve-self-signed-certs\""
Sep 12 03:03:02.229014 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.228989    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"]
Sep 12 03:03:02.405647 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:02.405822 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tls-certs\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.405822 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-model-cache\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.405903 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tmp-dir\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.405938 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-dshm\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.405977 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-home\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.406011 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.405996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpsg\" (UniqueName: \"kubernetes.io/projected/ab7d4f07-7341-4f89-85c2-8510b9d20f22-kube-api-access-4tpsg\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507085 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.506996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpsg\" (UniqueName: \"kubernetes.io/projected/ab7d4f07-7341-4f89-85c2-8510b9d20f22-kube-api-access-4tpsg\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507085 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tls-certs\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507085 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-model-cache\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507365 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tmp-dir\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507365 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-dshm\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507365 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-home\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507803 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tmp-dir\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.507999 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.507976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-model-cache\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.508168 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.508112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-home\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.509852 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.509827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tls-certs\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.509963 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.509830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-dshm\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.514711 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.514689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpsg\" (UniqueName: \"kubernetes.io/projected/ab7d4f07-7341-4f89-85c2-8510b9d20f22-kube-api-access-4tpsg\") pod \"distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") " pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.531382 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.531357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:02.811736 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:02.811710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"]
Sep 12 03:03:02.813261 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:03:02.813238    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab7d4f07_7341_4f89_85c2_8510b9d20f22.slice/crio-9d80112be636593d08ae24c8879e01a5fe0cfb5df3464fe145d7d4c42108db49 WatchSource:0}: Error finding container 9d80112be636593d08ae24c8879e01a5fe0cfb5df3464fe145d7d4c42108db49: Status 404 returned error can't find the container with id 9d80112be636593d08ae24c8879e01a5fe0cfb5df3464fe145d7d4c42108db49
Sep 12 03:03:03.551330 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:03.551293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn" event={"ID":"ab7d4f07-7341-4f89-85c2-8510b9d20f22","Type":"ContainerStarted","Data":"bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2"}
Sep 12 03:03:03.551330 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:03.551331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn" event={"ID":"ab7d4f07-7341-4f89-85c2-8510b9d20f22","Type":"ContainerStarted","Data":"9d80112be636593d08ae24c8879e01a5fe0cfb5df3464fe145d7d4c42108db49"}
Sep 12 03:03:03.567683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:03.567636    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn" podStartSLOduration=1.5676209220000001 podStartE2EDuration="1.567620922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:03:03.566628476 +0000 UTC m=+1601.849542055" watchObservedRunningTime="2026-09-12 03:03:03.567620922 +0000 UTC m=+1601.850534501"
Sep 12 03:03:12.531541 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:12.531509    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:12.532042 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:12.531565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:12.544599 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:12.544571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:12.609079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:12.609047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:03:20.509858 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:20.509830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:25.596132 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:25.596089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:26.922304 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:26.922274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:27.022121 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:27.022090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:27.314917 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:27.314830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:27.645709 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:27.645676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:27.795702 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:27.795667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:29.907923 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:29.907890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:30.574687 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:30.574657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:30.640228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:30.640179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:31.449634 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:31.449603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:32.218051 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:32.218021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:32.875493 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:32.875465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:33.682874 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:33.682841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:34.305498 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:34.305437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:34.954047 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:34.954002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:35.055874 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.055843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:35.153595 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.153560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:35.470975 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.470937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:35.562405 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.562363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:35.567962 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.567934    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"]
Sep 12 03:03:35.571626 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.571607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.574367 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.574349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-69d2db-dockercfg-xpjlk\""
Sep 12 03:03:35.574480 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.574348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-69d2db\""
Sep 12 03:03:35.581735 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.581710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"]
Sep 12 03:03:35.747207 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.747103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrtf\" (UniqueName: \"kubernetes.io/projected/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-kube-api-access-chrtf\") pod \"payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.747207 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.747197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.848156 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.848116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.848337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.848245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrtf\" (UniqueName: \"kubernetes.io/projected/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-kube-api-access-chrtf\") pod \"payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.848855 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.848836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.856392 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.856365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrtf\" (UniqueName: \"kubernetes.io/projected/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-kube-api-access-chrtf\") pod \"payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:35.883340 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:35.883312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:36.155477 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:36.155437    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"]
Sep 12 03:03:36.157532 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:03:36.157493    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ec080c_43b7_4907_b985_b8deaa6ce6fd.slice/crio-447b3402fff821e017ca3c2ff5cac619e97983b1dd7502fef75f3eb8e88c87b9 WatchSource:0}: Error finding container 447b3402fff821e017ca3c2ff5cac619e97983b1dd7502fef75f3eb8e88c87b9: Status 404 returned error can't find the container with id 447b3402fff821e017ca3c2ff5cac619e97983b1dd7502fef75f3eb8e88c87b9
Sep 12 03:03:36.728913 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:36.728860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw" event={"ID":"f5ec080c-43b7-4907-b985-b8deaa6ce6fd","Type":"ContainerStarted","Data":"b3bf5b5c2a2a47081f9ab436ecd913d4c472da10e289fb375d7481fd4ad804f5"}
Sep 12 03:03:36.729314 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:36.728935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw" event={"ID":"f5ec080c-43b7-4907-b985-b8deaa6ce6fd","Type":"ContainerStarted","Data":"447b3402fff821e017ca3c2ff5cac619e97983b1dd7502fef75f3eb8e88c87b9"}
Sep 12 03:03:36.729314 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:36.728963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:36.747231 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:36.747183    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw" podStartSLOduration=1.747168072 podStartE2EDuration="1.747168072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:03:36.745804113 +0000 UTC m=+1635.028717694" watchObservedRunningTime="2026-09-12 03:03:36.747168072 +0000 UTC m=+1635.030081653"
Sep 12 03:03:47.736385 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:47.736356    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:03:56.205575 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:56.205538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:57.154056 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:57.154016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:57.956284 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:57.956253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:58.721530 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:58.721495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:03:59.524189 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:03:59.524158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:00.466994 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:00.466966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:00.730128 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:00.730037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:01.184913 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.184872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"]
Sep 12 03:04:01.191496 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.191472    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"]
Sep 12 03:04:01.209062 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.209034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:01.325945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.325910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:01.424167 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.424135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:01.519239 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.519163    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"]
Sep 12 03:04:01.522916 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.522897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.526853 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.526832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-abddb9-openshift-default-dockercfg-glnb5\""
Sep 12 03:04:01.536564 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.536538    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"]
Sep 12 03:04:01.553249 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.553226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:01.588414 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-data\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588602 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-certs\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588602 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-credential-socket\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588728 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-kube-api-access-xtfjw\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588728 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-socket\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588833 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-podinfo\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588833 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-envoy\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588922 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istiod-ca-cert\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.588961 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.588928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-token\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.689636 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-kube-api-access-xtfjw\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.689825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-socket\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.689825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-podinfo\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.689825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-envoy\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.689825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istiod-ca-cert\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690045 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-token\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690045 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-data\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690045 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-certs\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690045 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.689982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-credential-socket\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690253 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.690136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-socket\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690253 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.690228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-data\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690384 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.690359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-credential-socket\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690600 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.690511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-certs\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.690600 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.690633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istiod-ca-cert\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.692180 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.692161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-envoy\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.692394 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.692376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-podinfo\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.698122 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.698098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-token\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.698248 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.698235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-kube-api-access-xtfjw\") pod \"e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") " pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:01.834074 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:01.833990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:02.001387 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.001349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:02.119770 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.119743    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"]
Sep 12 03:04:02.122370 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:04:02.122344    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5b6443_0d70_4ab5_84b1_ec1a12987d65.slice/crio-d815a671db59d334e4582f2c23a43f2394a052c5636d8d5e42044530d2614d85 WatchSource:0}: Error finding container d815a671db59d334e4582f2c23a43f2394a052c5636d8d5e42044530d2614d85: Status 404 returned error can't find the container with id d815a671db59d334e4582f2c23a43f2394a052c5636d8d5e42044530d2614d85
Sep 12 03:04:02.124697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.124657    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:04:02.124817 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.124718    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:04:02.124817 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.124745    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:04:02.196616 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.196594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:02.869386 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.869346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" event={"ID":"1d5b6443-0d70-4ab5-84b1-ec1a12987d65","Type":"ContainerStarted","Data":"87e112071f65504090a9d0a60e6c59725d3742f02a5d015e78ab25a7774f0874"}
Sep 12 03:04:02.869866 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.869409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" event={"ID":"1d5b6443-0d70-4ab5-84b1-ec1a12987d65","Type":"ContainerStarted","Data":"d815a671db59d334e4582f2c23a43f2394a052c5636d8d5e42044530d2614d85"}
Sep 12 03:04:02.895263 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:02.895211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podStartSLOduration=1.8951929760000001 podStartE2EDuration="1.895192976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:04:02.893854069 +0000 UTC m=+1661.176767649" watchObservedRunningTime="2026-09-12 03:04:02.895192976 +0000 UTC m=+1661.178106558"
Sep 12 03:04:03.147735 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:03.147622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:03.834559 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:03.834519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:03.839467 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:03.839421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:03.840875 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:03.840853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:03.874408 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:03.874380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:03.875306 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:03.875288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:04.679944 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:04.679916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:04.859019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:04.858983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:04.965253 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:04.965181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:05.559979 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:05.559948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:06.445362 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:06.445332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:07.169505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:07.169476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:07.913846 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:07.913813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:08.671230 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:08.671196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:09.345869 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:09.345835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:10.078107 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:10.078074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:10.774116 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:10.774080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:10.874697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:10.874652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:10.974505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:10.974473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:11.229563 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:11.229530    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:11.330539 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:11.330505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:16.208371 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.208334    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf" podUID="ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" containerName="main" containerID="cri-o://4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b" gracePeriod=45
Sep 12 03:04:16.219914 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.219879    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p" podUID="5c90f719-277e-4f8a-b672-046d026cdd3c" containerName="main" containerID="cri-o://6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b" gracePeriod=45
Sep 12 03:04:16.467046 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.466971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 03:04:16.509146 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.509115    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 03:04:16.558775 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.558742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"]
Sep 12 03:04:16.558966 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.558939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62" podUID="067a47dc-2c59-4898-bf89-5b8063f5f90a" containerName="maas-api" containerID="cri-o://d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83" gracePeriod=30
Sep 12 03:04:16.599711 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.599676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:16.651251 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-dshm\") pod \"5c90f719-277e-4f8a-b672-046d026cdd3c\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") "
Sep 12 03:04:16.651416 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qr4p\" (UniqueName: \"kubernetes.io/projected/5c90f719-277e-4f8a-b672-046d026cdd3c-kube-api-access-2qr4p\") pod \"5c90f719-277e-4f8a-b672-046d026cdd3c\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") "
Sep 12 03:04:16.651416 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-model-cache\") pod \"5c90f719-277e-4f8a-b672-046d026cdd3c\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") "
Sep 12 03:04:16.651416 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-model-cache\") pod \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") "
Sep 12 03:04:16.651416 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tmp-dir\") pod \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-tmp-dir\") pod \"5c90f719-277e-4f8a-b672-046d026cdd3c\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-home\") pod \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-dshm\") pod \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-home\") pod \"5c90f719-277e-4f8a-b672-046d026cdd3c\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74jp5\" (UniqueName: \"kubernetes.io/projected/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-kube-api-access-74jp5\") pod \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651621    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c90f719-277e-4f8a-b672-046d026cdd3c-tls-certs\") pod \"5c90f719-277e-4f8a-b672-046d026cdd3c\" (UID: \"5c90f719-277e-4f8a-b672-046d026cdd3c\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651637    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c90f719-277e-4f8a-b672-046d026cdd3c" (UID: "5c90f719-277e-4f8a-b672-046d026cdd3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tls-certs\") pod \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\" (UID: \"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431\") "
Sep 12 03:04:16.651664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" (UID: "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.652205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-home" (OuterVolumeSpecName: "home") pod "5c90f719-277e-4f8a-b672-046d026cdd3c" (UID: "5c90f719-277e-4f8a-b672-046d026cdd3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.652205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.651879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c90f719-277e-4f8a-b672-046d026cdd3c" (UID: "5c90f719-277e-4f8a-b672-046d026cdd3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.652205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.652033    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.652205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.652052    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.652205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.652066    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.652205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.652103    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.652707 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.652066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-home" (OuterVolumeSpecName: "home") pod "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" (UID: "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.652707 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.652385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" (UID: "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.654788 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.654752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-dshm" (OuterVolumeSpecName: "dshm") pod "5c90f719-277e-4f8a-b672-046d026cdd3c" (UID: "5c90f719-277e-4f8a-b672-046d026cdd3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.654788 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.654756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c90f719-277e-4f8a-b672-046d026cdd3c-kube-api-access-2qr4p" (OuterVolumeSpecName: "kube-api-access-2qr4p") pod "5c90f719-277e-4f8a-b672-046d026cdd3c" (UID: "5c90f719-277e-4f8a-b672-046d026cdd3c"). InnerVolumeSpecName "kube-api-access-2qr4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:16.654981 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.654790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-dshm" (OuterVolumeSpecName: "dshm") pod "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" (UID: "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:16.655047 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.655000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c90f719-277e-4f8a-b672-046d026cdd3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c90f719-277e-4f8a-b672-046d026cdd3c" (UID: "5c90f719-277e-4f8a-b672-046d026cdd3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.655105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.655059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" (UID: "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.655105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.655069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-kube-api-access-74jp5" (OuterVolumeSpecName: "kube-api-access-74jp5") pod "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" (UID: "ee7e2bb2-01fe-41bb-a3e2-04f25ea53431"). InnerVolumeSpecName "kube-api-access-74jp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:16.669397 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.668136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"]
Sep 12 03:04:16.669397 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.668392    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6" podUID="9de4c729-3351-428c-8cc3-78ed7cd2c751" containerName="payload-processing" containerID="cri-o://1ca659c7223c850d6f404f8810b4ed7938728f96ba3c60bf55855045d43ef84e" gracePeriod=30
Sep 12 03:04:16.718018 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.717907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753228    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753264    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c90f719-277e-4f8a-b672-046d026cdd3c-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753279    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qr4p\" (UniqueName: \"kubernetes.io/projected/5c90f719-277e-4f8a-b672-046d026cdd3c-kube-api-access-2qr4p\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753296    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753311    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753321    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753333    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jp5\" (UniqueName: \"kubernetes.io/projected/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431-kube-api-access-74jp5\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.753347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.753345    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c90f719-277e-4f8a-b672-046d026cdd3c-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.817466 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.817412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:16.820326 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.820303    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 03:04:16.924619 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.924590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:16.952673 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.952639    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" containerID="4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b" exitCode=0
Sep 12 03:04:16.952847 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.952675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf" event={"ID":"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431","Type":"ContainerDied","Data":"4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b"}
Sep 12 03:04:16.952847 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.952718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf" event={"ID":"ee7e2bb2-01fe-41bb-a3e2-04f25ea53431","Type":"ContainerDied","Data":"7399356ea8c629d5340b71d26e9da9f713645db207bc545001b29ff2b0fc4089"}
Sep 12 03:04:16.952847 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.952726    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"
Sep 12 03:04:16.952847 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.952742    2579 scope.go:117] "RemoveContainer" containerID="4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b"
Sep 12 03:04:16.954337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067a47dc-2c59-4898-bf89-5b8063f5f90a-openshift-service-ca\") pod \"067a47dc-2c59-4898-bf89-5b8063f5f90a\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") "
Sep 12 03:04:16.954407 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-metrics-certs\") pod \"067a47dc-2c59-4898-bf89-5b8063f5f90a\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") "
Sep 12 03:04:16.954499 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9sch\" (UniqueName: \"kubernetes.io/projected/067a47dc-2c59-4898-bf89-5b8063f5f90a-kube-api-access-s9sch\") pod \"067a47dc-2c59-4898-bf89-5b8063f5f90a\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") "
Sep 12 03:04:16.954594 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-maas-api-tls\") pod \"067a47dc-2c59-4898-bf89-5b8063f5f90a\" (UID: \"067a47dc-2c59-4898-bf89-5b8063f5f90a\") "
Sep 12 03:04:16.954679 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954638    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067a47dc-2c59-4898-bf89-5b8063f5f90a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "067a47dc-2c59-4898-bf89-5b8063f5f90a" (UID: "067a47dc-2c59-4898-bf89-5b8063f5f90a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:04:16.954825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954803    2579 generic.go:356] "Generic (PLEG): container finished" podID="9de4c729-3351-428c-8cc3-78ed7cd2c751" containerID="1ca659c7223c850d6f404f8810b4ed7938728f96ba3c60bf55855045d43ef84e" exitCode=0
Sep 12 03:04:16.955037 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.955020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 03:04:16.955139 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6" event={"ID":"9de4c729-3351-428c-8cc3-78ed7cd2c751","Type":"ContainerDied","Data":"1ca659c7223c850d6f404f8810b4ed7938728f96ba3c60bf55855045d43ef84e"}
Sep 12 03:04:16.955236 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.954875    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/067a47dc-2c59-4898-bf89-5b8063f5f90a-openshift-service-ca\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:16.957424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.957395    2579 generic.go:356] "Generic (PLEG): container finished" podID="067a47dc-2c59-4898-bf89-5b8063f5f90a" containerID="d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83" exitCode=0
Sep 12 03:04:16.957550 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.957426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067a47dc-2c59-4898-bf89-5b8063f5f90a-kube-api-access-s9sch" (OuterVolumeSpecName: "kube-api-access-s9sch") pod "067a47dc-2c59-4898-bf89-5b8063f5f90a" (UID: "067a47dc-2c59-4898-bf89-5b8063f5f90a"). InnerVolumeSpecName "kube-api-access-s9sch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:16.957619 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.957556    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"
Sep 12 03:04:16.957619 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.957575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62" event={"ID":"067a47dc-2c59-4898-bf89-5b8063f5f90a","Type":"ContainerDied","Data":"d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83"}
Sep 12 03:04:16.957733 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.957619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62" event={"ID":"067a47dc-2c59-4898-bf89-5b8063f5f90a","Type":"ContainerDied","Data":"0b4eeb818a207042cd8f8d97c20faac59ccd829a36976671e24cfff4a8f863ef"}
Sep 12 03:04:16.957733 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.957695    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "067a47dc-2c59-4898-bf89-5b8063f5f90a" (UID: "067a47dc-2c59-4898-bf89-5b8063f5f90a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.958299 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.958273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "067a47dc-2c59-4898-bf89-5b8063f5f90a" (UID: "067a47dc-2c59-4898-bf89-5b8063f5f90a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:04:16.959861 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.959835    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c90f719-277e-4f8a-b672-046d026cdd3c" containerID="6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b" exitCode=0
Sep 12 03:04:16.959982 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.959964    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"
Sep 12 03:04:16.960051 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.959995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p" event={"ID":"5c90f719-277e-4f8a-b672-046d026cdd3c","Type":"ContainerDied","Data":"6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b"}
Sep 12 03:04:16.960051 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.960028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p" event={"ID":"5c90f719-277e-4f8a-b672-046d026cdd3c","Type":"ContainerDied","Data":"2a00fb3c8661098b1dfff10c66fd39797c5048cc31122432f7856520e740a154"}
Sep 12 03:04:16.976208 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.975674    2579 scope.go:117] "RemoveContainer" containerID="4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b"
Sep 12 03:04:16.976328 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:04:16.976262    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b\": container with ID starting with 4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b not found: ID does not exist" containerID="4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b"
Sep 12 03:04:16.976328 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.976291    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b"} err="failed to get container status \"4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b\": rpc error: code = NotFound desc = could not find container \"4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b\": container with ID starting with 4a1bce822d652176d9f66e8e4991085f010f69346e15969195dff5b5b003ff1b not found: ID does not exist"
Sep 12 03:04:16.976328 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.976312    2579 scope.go:117] "RemoveContainer" containerID="1ca659c7223c850d6f404f8810b4ed7938728f96ba3c60bf55855045d43ef84e"
Sep 12 03:04:16.989111 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.989091    2579 scope.go:117] "RemoveContainer" containerID="d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83"
Sep 12 03:04:16.999998 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:16.999976    2579 scope.go:117] "RemoveContainer" containerID="d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83"
Sep 12 03:04:17.000494 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:04:17.000472    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83\": container with ID starting with d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83 not found: ID does not exist" containerID="d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83"
Sep 12 03:04:17.000635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.000501    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83"} err="failed to get container status \"d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83\": rpc error: code = NotFound desc = could not find container \"d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83\": container with ID starting with d809e60e1f73b4fb1162574e0955a82184dc8b47c679947c9d562448ac436e83 not found: ID does not exist"
Sep 12 03:04:17.000635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.000520    2579 scope.go:117] "RemoveContainer" containerID="6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b"
Sep 12 03:04:17.002171 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.002144    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"]
Sep 12 03:04:17.012157 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.012133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:17.012416 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.012342    2579 scope.go:117] "RemoveContainer" containerID="6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b"
Sep 12 03:04:17.012743 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:04:17.012720    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b\": container with ID starting with 6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b not found: ID does not exist" containerID="6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b"
Sep 12 03:04:17.012827 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.012749    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b"} err="failed to get container status \"6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b\": rpc error: code = NotFound desc = could not find container \"6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b\": container with ID starting with 6c8fce0d7383aa3480c176a2a911097cd121c90f2dd56ca253326e87d212525b not found: ID does not exist"
Sep 12 03:04:17.013728 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.013704    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-2-w0-b0573c-kserve-868759699b-97fkf"]
Sep 12 03:04:17.014822 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:04:17.014797    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7e2bb2_01fe_41bb_a3e2_04f25ea53431.slice/crio-7399356ea8c629d5340b71d26e9da9f713645db207bc545001b29ff2b0fc4089\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c90f719_277e_4f8a_b672_046d026cdd3c.slice/crio-2a00fb3c8661098b1dfff10c66fd39797c5048cc31122432f7856520e740a154\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c90f719_277e_4f8a_b672_046d026cdd3c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7e2bb2_01fe_41bb_a3e2_04f25ea53431.slice\": RecentStats: unable to find data in memory cache]"
Sep 12 03:04:17.027509 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.027483    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"]
Sep 12 03:04:17.031429 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.031406    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b0573c/distinct-w0-b0573c-kserve-748bd8fc59-tqp9p"]
Sep 12 03:04:17.055728 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.055696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lkfw\" (UniqueName: \"kubernetes.io/projected/9de4c729-3351-428c-8cc3-78ed7cd2c751-kube-api-access-7lkfw\") pod \"9de4c729-3351-428c-8cc3-78ed7cd2c751\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") "
Sep 12 03:04:17.055904 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.055790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de4c729-3351-428c-8cc3-78ed7cd2c751-plugins-config-volume\") pod \"9de4c729-3351-428c-8cc3-78ed7cd2c751\" (UID: \"9de4c729-3351-428c-8cc3-78ed7cd2c751\") "
Sep 12 03:04:17.055991 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.055980    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-maas-api-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:17.056044 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.056004    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/067a47dc-2c59-4898-bf89-5b8063f5f90a-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:17.056044 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.056018    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9sch\" (UniqueName: \"kubernetes.io/projected/067a47dc-2c59-4898-bf89-5b8063f5f90a-kube-api-access-s9sch\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:17.056255 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.056228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de4c729-3351-428c-8cc3-78ed7cd2c751-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9de4c729-3351-428c-8cc3-78ed7cd2c751" (UID: "9de4c729-3351-428c-8cc3-78ed7cd2c751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:04:17.058058 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.058036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de4c729-3351-428c-8cc3-78ed7cd2c751-kube-api-access-7lkfw" (OuterVolumeSpecName: "kube-api-access-7lkfw") pod "9de4c729-3351-428c-8cc3-78ed7cd2c751" (UID: "9de4c729-3351-428c-8cc3-78ed7cd2c751"). InnerVolumeSpecName "kube-api-access-7lkfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:17.156928 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.156882    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lkfw\" (UniqueName: \"kubernetes.io/projected/9de4c729-3351-428c-8cc3-78ed7cd2c751-kube-api-access-7lkfw\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:17.156928 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.156921    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de4c729-3351-428c-8cc3-78ed7cd2c751-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:17.283819 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.283776    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"]
Sep 12 03:04:17.289018 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.288993    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b0573c-f844578cb-6nt62"]
Sep 12 03:04:17.310174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.310146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:17.466571 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.466546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:17.967745 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.967713    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"
Sep 12 03:04:17.967745 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.967726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6" event={"ID":"9de4c729-3351-428c-8cc3-78ed7cd2c751","Type":"ContainerDied","Data":"deb342baff9c68db63f19ca8761dfc028c67c61a6d6fdb44dafacc174a86f25a"}
Sep 12 03:04:17.994834 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:17.994805    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"]
Sep 12 03:04:18.004228 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:18.004206    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b0573c-7668795bcf-lckz6"]
Sep 12 03:04:18.247692 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:18.247610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067a47dc-2c59-4898-bf89-5b8063f5f90a" path="/var/lib/kubelet/pods/067a47dc-2c59-4898-bf89-5b8063f5f90a/volumes"
Sep 12 03:04:18.248064 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:18.248047    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c90f719-277e-4f8a-b672-046d026cdd3c" path="/var/lib/kubelet/pods/5c90f719-277e-4f8a-b672-046d026cdd3c/volumes"
Sep 12 03:04:18.248363 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:18.248349    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de4c729-3351-428c-8cc3-78ed7cd2c751" path="/var/lib/kubelet/pods/9de4c729-3351-428c-8cc3-78ed7cd2c751/volumes"
Sep 12 03:04:18.249025 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:18.249003    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" path="/var/lib/kubelet/pods/ee7e2bb2-01fe-41bb-a3e2-04f25ea53431/volumes"
Sep 12 03:04:18.435994 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:18.435963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:19.575252 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:19.575222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:20.341710 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:20.341681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:21.354802 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:21.354771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:22.265800 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:22.265755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:23.154812 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:23.154778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:23.830622 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:23.830586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:24.404999 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:24.404967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:24.500635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:24.500601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:24.694430 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:24.694336    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"]
Sep 12 03:04:24.694612 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:24.694585    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy" containerID="cri-o://d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361" gracePeriod=30
Sep 12 03:04:24.714494 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:24.714462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:25.569338 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:25.569307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:26.607433 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:26.607405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:27.534586 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:27.534552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:28.710632 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:28.710600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:29.783715 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.783681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:29.894220 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.894184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:29.957887 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.957866    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 03:04:29.978872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.978799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-token\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.978872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.978858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574e6656-2c15-44ef-8f77-27a10fc503b3-istiod-ca-cert\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979068 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.978890    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-data\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979068 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.978955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trls4\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-kube-api-access-trls4\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979068 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.978991    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-certs\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979223 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-podinfo\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979223 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-credential-socket\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979322 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979233    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-envoy\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979322 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-socket\") pod \"574e6656-2c15-44ef-8f77-27a10fc503b3\" (UID: \"574e6656-2c15-44ef-8f77-27a10fc503b3\") "
Sep 12 03:04:29.979414 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:29.979570 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:29.979887 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979620    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:29.979887 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979641    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:29.979887 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:29.979887 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979844    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574e6656-2c15-44ef-8f77-27a10fc503b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:04:29.980124 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.979914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:29.981344 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.981313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:29.981431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.981373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-kube-api-access-trls4" (OuterVolumeSpecName: "kube-api-access-trls4") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "kube-api-access-trls4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:29.981902 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.981880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:04:29.981966 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:29.981910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "574e6656-2c15-44ef-8f77-27a10fc503b3" (UID: "574e6656-2c15-44ef-8f77-27a10fc503b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:04:30.035003 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.034964    2579 generic.go:356] "Generic (PLEG): container finished" podID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerID="d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361" exitCode=0
Sep 12 03:04:30.035135 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.035010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" event={"ID":"574e6656-2c15-44ef-8f77-27a10fc503b3","Type":"ContainerDied","Data":"d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361"}
Sep 12 03:04:30.035135 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.035024    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"
Sep 12 03:04:30.035135 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.035052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz" event={"ID":"574e6656-2c15-44ef-8f77-27a10fc503b3","Type":"ContainerDied","Data":"497f39b914eef2cd4f9b64d340082c62fbb63415f5e1bc1d31eaa4910dae6a88"}
Sep 12 03:04:30.035135 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.035073    2579 scope.go:117] "RemoveContainer" containerID="d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361"
Sep 12 03:04:30.044540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.044526    2579 scope.go:117] "RemoveContainer" containerID="9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"
Sep 12 03:04:30.052498 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.052482    2579 scope.go:117] "RemoveContainer" containerID="d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361"
Sep 12 03:04:30.052754 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:04:30.052733    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361\": container with ID starting with d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361 not found: ID does not exist" containerID="d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361"
Sep 12 03:04:30.052797 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.052763    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361"} err="failed to get container status \"d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361\": rpc error: code = NotFound desc = could not find container \"d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361\": container with ID starting with d5a1989d74ad99f52ba2ae34151360e5a06e19526f3d5aaf50b9a8f903bd9361 not found: ID does not exist"
Sep 12 03:04:30.052797 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.052780    2579 scope.go:117] "RemoveContainer" containerID="9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"
Sep 12 03:04:30.053055 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:04:30.053037    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640\": container with ID starting with 9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640 not found: ID does not exist" containerID="9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"
Sep 12 03:04:30.053114 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.053063    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640"} err="failed to get container status \"9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640\": rpc error: code = NotFound desc = could not find container \"9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640\": container with ID starting with 9fbd6f3e26562445eed8ee50b77e6cc0c10fc58c65a41cd6282ee347f8f71640 not found: ID does not exist"
Sep 12 03:04:30.058601 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.058578    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"]
Sep 12 03:04:30.061557 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.061537    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b0573c-openshift-default-77dc684668-zclkz"]
Sep 12 03:04:30.080697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080673    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.080697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080696    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.080813 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080707    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.080813 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080715    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.080813 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080722    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574e6656-2c15-44ef-8f77-27a10fc503b3-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.080813 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080729    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trls4\" (UniqueName: \"kubernetes.io/projected/574e6656-2c15-44ef-8f77-27a10fc503b3-kube-api-access-trls4\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.080813 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.080737    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574e6656-2c15-44ef-8f77-27a10fc503b3-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:30.245604 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.245532    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" path="/var/lib/kubelet/pods/574e6656-2c15-44ef-8f77-27a10fc503b3/volumes"
Sep 12 03:04:30.749112 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:30.749080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:31.524699 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:31.524665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:32.444416 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:32.444386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:33.349085 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:33.349057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:34.125509 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:34.125479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:35.053467 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:35.053419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:35.934787 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:35.934748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:36.671237 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:36.671206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:37.549658 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:37.549622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:38.099605 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:38.099570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:38.199826 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:38.199794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:38.397542 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:38.397501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:38.874933 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:38.874896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:38.973355 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:38.973327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:39.129078 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:39.128999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:39.459090 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:39.459008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:40.854996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:40.854965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:43.165945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:43.165863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:44.608854 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:44.608818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:48.865772 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:48.865740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:48.887243 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:48.887211    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"]
Sep 12 03:04:48.887509 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:48.887460    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw" podUID="f5ec080c-43b7-4907-b985-b8deaa6ce6fd" containerName="payload-pre-processing" containerID="cri-o://b3bf5b5c2a2a47081f9ab436ecd913d4c472da10e289fb375d7481fd4ad804f5" gracePeriod=30
Sep 12 03:04:48.965347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:48.965308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:49.044176 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.044144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:49.106221 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.106017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:49.155263 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.155196    2579 generic.go:356] "Generic (PLEG): container finished" podID="f5ec080c-43b7-4907-b985-b8deaa6ce6fd" containerID="b3bf5b5c2a2a47081f9ab436ecd913d4c472da10e289fb375d7481fd4ad804f5" exitCode=0
Sep 12 03:04:49.155263 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.155227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw" event={"ID":"f5ec080c-43b7-4907-b985-b8deaa6ce6fd","Type":"ContainerDied","Data":"b3bf5b5c2a2a47081f9ab436ecd913d4c472da10e289fb375d7481fd4ad804f5"}
Sep 12 03:04:49.155442 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.155290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw" event={"ID":"f5ec080c-43b7-4907-b985-b8deaa6ce6fd","Type":"ContainerDied","Data":"447b3402fff821e017ca3c2ff5cac619e97983b1dd7502fef75f3eb8e88c87b9"}
Sep 12 03:04:49.155442 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.155317    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="447b3402fff821e017ca3c2ff5cac619e97983b1dd7502fef75f3eb8e88c87b9"
Sep 12 03:04:49.158690 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.158666    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:04:49.265504 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.265438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chrtf\" (UniqueName: \"kubernetes.io/projected/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-kube-api-access-chrtf\") pod \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") "
Sep 12 03:04:49.265682 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.265572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-plugins-config-volume\") pod \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\" (UID: \"f5ec080c-43b7-4907-b985-b8deaa6ce6fd\") "
Sep 12 03:04:49.265974 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.265953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5ec080c-43b7-4907-b985-b8deaa6ce6fd" (UID: "f5ec080c-43b7-4907-b985-b8deaa6ce6fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:04:49.267787 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.267759    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-kube-api-access-chrtf" (OuterVolumeSpecName: "kube-api-access-chrtf") pod "f5ec080c-43b7-4907-b985-b8deaa6ce6fd" (UID: "f5ec080c-43b7-4907-b985-b8deaa6ce6fd"). InnerVolumeSpecName "kube-api-access-chrtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:04:49.367789 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.367409    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:49.367789 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.367440    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrtf\" (UniqueName: \"kubernetes.io/projected/f5ec080c-43b7-4907-b985-b8deaa6ce6fd-kube-api-access-chrtf\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:04:49.431733 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.431646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:49.499382 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.499354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:49.916630 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.916590    2579 patch_prober.go:28] interesting pod/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 03:04:49.917062 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:49.916667    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 03:04:50.156664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:50.156630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:50.161849 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:50.161822    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"
Sep 12 03:04:50.185814 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:50.185782    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"]
Sep 12 03:04:50.195514 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:50.192506    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69d2db-86cbdd4bb6-lp4vw"]
Sep 12 03:04:50.246206 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:50.246174    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ec080c-43b7-4907-b985-b8deaa6ce6fd" path="/var/lib/kubelet/pods/f5ec080c-43b7-4907-b985-b8deaa6ce6fd/volumes"
Sep 12 03:04:50.952017 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:50.951984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:51.168944 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.168905    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerID="87e112071f65504090a9d0a60e6c59725d3742f02a5d015e78ab25a7774f0874" exitCode=0
Sep 12 03:04:51.169099 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.168977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" event={"ID":"1d5b6443-0d70-4ab5-84b1-ec1a12987d65","Type":"ContainerDied","Data":"87e112071f65504090a9d0a60e6c59725d3742f02a5d015e78ab25a7774f0874"}
Sep 12 03:04:51.169394 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.169379    2579 scope.go:117] "RemoveContainer" containerID="87e112071f65504090a9d0a60e6c59725d3742f02a5d015e78ab25a7774f0874"
Sep 12 03:04:51.170495 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.170472    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:04:51.170557 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.170535    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:04:51.170597 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.170564    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:04:51.657639 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:51.657602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:52.193835 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:52.193802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" event={"ID":"1d5b6443-0d70-4ab5-84b1-ec1a12987d65","Type":"ContainerStarted","Data":"343eaa79e17f3a7a739c9b2f9ac5f357a2d86390afe24b0cf90685322c39a4dd"}
Sep 12 03:04:52.356441 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:52.356414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:52.835013 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:52.834985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:52.835205 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:52.835185    2579 patch_prober.go:28] interesting pod/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 03:04:52.835257 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:52.835237    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 03:04:53.183217 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:53.183122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:53.834353 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:53.834317    2579 patch_prober.go:28] interesting pod/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 03:04:53.834749 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:53.834376    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 03:04:53.892145 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:53.892092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:54.834618 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:54.834579    2579 patch_prober.go:28] interesting pod/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 03:04:54.835012 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:54.834642    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 03:04:55.179614 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:55.179571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:55.279959 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:55.279842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:55.469268 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:55.469184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:55.834600 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:55.834518    2579 patch_prober.go:28] interesting pod/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 03:04:55.834743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:55.834594    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 03:04:56.834796 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:56.834760    2579 patch_prober.go:28] interesting pod/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 03:04:56.835201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:56.834816    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 03:04:56.862504 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:56.862406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:57.541890 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:57.541859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:57.838670 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:57.838586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:57.839046 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:57.838984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:57.839696 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:57.839678    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:04:58.206171 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:58.206123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:59.068964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:59.068936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:04:59.825937 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:04:59.825911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:00.569107 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:00.569081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:01.227223 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:01.227190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:01.958538 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:01.958501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:06.311496 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:06.311465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:06.599632 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:06.599499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:11.577510 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:11.577472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:11.645115 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:11.645085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:11.744312 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:11.744283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:11.797423 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:11.797393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:12.187635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:12.187599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:12.288665 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:12.288633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:13.010633 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:13.010602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:13.657638 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:13.657606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:14.416397 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:14.416361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:15.068714 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:15.068683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:15.747431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:15.747402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:17.773274 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:17.773203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:17.873522 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:17.873491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:17.930351 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:17.930300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:18.129750 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:18.129720    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"]
Sep 12 03:05:18.130007 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:18.129979    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy" containerID="cri-o://343eaa79e17f3a7a739c9b2f9ac5f357a2d86390afe24b0cf90685322c39a4dd" gracePeriod=30
Sep 12 03:05:18.152828 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:18.152795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:18.844418 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:18.844365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:19.745232 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:19.745203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:20.490546 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:20.490509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:21.239439 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:21.239408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:21.970185 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:21.970138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:22.706704 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:22.706580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:23.355709 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.355677    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerID="343eaa79e17f3a7a739c9b2f9ac5f357a2d86390afe24b0cf90685322c39a4dd" exitCode=0
Sep 12 03:05:23.356074 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.355755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" event={"ID":"1d5b6443-0d70-4ab5-84b1-ec1a12987d65","Type":"ContainerDied","Data":"343eaa79e17f3a7a739c9b2f9ac5f357a2d86390afe24b0cf90685322c39a4dd"}
Sep 12 03:05:23.356074 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.355827    2579 scope.go:117] "RemoveContainer" containerID="87e112071f65504090a9d0a60e6c59725d3742f02a5d015e78ab25a7774f0874"
Sep 12 03:05:23.400266 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.400210    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:05:23.471857 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.471826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:23.487605 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-socket\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487763 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-token\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487763 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-podinfo\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487763 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-data\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487947 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-kube-api-access-xtfjw\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487947 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-envoy\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487947 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-credential-socket\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487947 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-certs\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.487947 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istiod-ca-cert\") pod \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\" (UID: \"1d5b6443-0d70-4ab5-84b1-ec1a12987d65\") "
Sep 12 03:05:23.488213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.487839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:23.488213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.488072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:23.488327 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.488262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:23.488327 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.488274    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.488327 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.488292    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.488598 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.488321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:23.488598 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.488359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:05:23.490278 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.490243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:23.490384 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.490341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-kube-api-access-xtfjw" (OuterVolumeSpecName: "kube-api-access-xtfjw") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "kube-api-access-xtfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:23.490480 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.490433    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:23.490547 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.490530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d5b6443-0d70-4ab5-84b1-ec1a12987d65" (UID: "1d5b6443-0d70-4ab5-84b1-ec1a12987d65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:05:23.589364 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589328    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.589364 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589365    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.589641 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589378    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-kube-api-access-xtfjw\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.589641 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589389    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.589641 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589397    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.589641 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589407    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:23.589641 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:23.589415    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d5b6443-0d70-4ab5-84b1-ec1a12987d65-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:24.362818 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:24.362789    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"
Sep 12 03:05:24.363248 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:24.362816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w" event={"ID":"1d5b6443-0d70-4ab5-84b1-ec1a12987d65","Type":"ContainerDied","Data":"d815a671db59d334e4582f2c23a43f2394a052c5636d8d5e42044530d2614d85"}
Sep 12 03:05:24.363248 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:24.362874    2579 scope.go:117] "RemoveContainer" containerID="343eaa79e17f3a7a739c9b2f9ac5f357a2d86390afe24b0cf90685322c39a4dd"
Sep 12 03:05:24.381140 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:24.381114    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"]
Sep 12 03:05:24.384584 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:24.384561    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-abddb9-openshift-default-676ccc9fc8-bkv8w"]
Sep 12 03:05:26.246062 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:26.246034    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" path="/var/lib/kubelet/pods/1d5b6443-0d70-4ab5-84b1-ec1a12987d65/volumes"
Sep 12 03:05:29.135765 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:29.135729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:29.204903 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:29.204868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:29.303224 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:29.303193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:29.382149 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:29.382121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:29.689179 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:29.689144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:29.790688 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:29.790654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:30.376830 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:30.376776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:31.034005 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:31.033976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:31.839270 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:31.839227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:32.500310 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:32.500229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:35.730692 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:35.730652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:35.783637 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:35.783609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:35.989399 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:35.989319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:36.655414 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:36.655383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:37.453417 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:37.453385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:38.102282 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:38.102251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:38.832420 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:38.832384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:39.476526 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:39.476496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:40.291671 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:40.291645    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:44.342207 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:44.342175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:45.982678 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:45.982649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:46.843068 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:46.843038    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"]
Sep 12 03:05:46.843260 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:46.843241    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" podUID="eed29cc7-8ac8-45dc-a650-409b2ec424dc" containerName="maas-api" containerID="cri-o://83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609" gracePeriod=30
Sep 12 03:05:46.876535 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:46.876505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:46.957025 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:46.956998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:47.026195 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.026161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:47.106458 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.106411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:47.116849 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.116829    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 03:05:47.221107 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.221077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed29cc7-8ac8-45dc-a650-409b2ec424dc-openshift-service-ca\") pod \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") "
Sep 12 03:05:47.221270 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.221133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhwfv\" (UniqueName: \"kubernetes.io/projected/eed29cc7-8ac8-45dc-a650-409b2ec424dc-kube-api-access-fhwfv\") pod \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") "
Sep 12 03:05:47.221270 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.221161    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-metrics-certs\") pod \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") "
Sep 12 03:05:47.221270 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.221183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls\") pod \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\" (UID: \"eed29cc7-8ac8-45dc-a650-409b2ec424dc\") "
Sep 12 03:05:47.221537 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.221504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed29cc7-8ac8-45dc-a650-409b2ec424dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eed29cc7-8ac8-45dc-a650-409b2ec424dc" (UID: "eed29cc7-8ac8-45dc-a650-409b2ec424dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:05:47.223496 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.223469    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eed29cc7-8ac8-45dc-a650-409b2ec424dc" (UID: "eed29cc7-8ac8-45dc-a650-409b2ec424dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:05:47.223594 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.223520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed29cc7-8ac8-45dc-a650-409b2ec424dc-kube-api-access-fhwfv" (OuterVolumeSpecName: "kube-api-access-fhwfv") pod "eed29cc7-8ac8-45dc-a650-409b2ec424dc" (UID: "eed29cc7-8ac8-45dc-a650-409b2ec424dc"). InnerVolumeSpecName "kube-api-access-fhwfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:47.223692 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.223673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eed29cc7-8ac8-45dc-a650-409b2ec424dc" (UID: "eed29cc7-8ac8-45dc-a650-409b2ec424dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:05:47.328609 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.322910    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed29cc7-8ac8-45dc-a650-409b2ec424dc-openshift-service-ca\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:47.328609 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.322945    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhwfv\" (UniqueName: \"kubernetes.io/projected/eed29cc7-8ac8-45dc-a650-409b2ec424dc-kube-api-access-fhwfv\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:47.328609 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.322962    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:47.328609 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.322984    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed29cc7-8ac8-45dc-a650-409b2ec424dc-maas-api-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:47.476845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.476817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:47.485705 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.485676    2579 generic.go:356] "Generic (PLEG): container finished" podID="eed29cc7-8ac8-45dc-a650-409b2ec424dc" containerID="83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609" exitCode=0
Sep 12 03:05:47.485871 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.485734    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"
Sep 12 03:05:47.485871 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.485757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" event={"ID":"eed29cc7-8ac8-45dc-a650-409b2ec424dc","Type":"ContainerDied","Data":"83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609"}
Sep 12 03:05:47.485871 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.485830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj" event={"ID":"eed29cc7-8ac8-45dc-a650-409b2ec424dc","Type":"ContainerDied","Data":"cf122f196ed722824a1b51bd4822b56957af09374124f9da83c6065c4f5c5f68"}
Sep 12 03:05:47.485871 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.485854    2579 scope.go:117] "RemoveContainer" containerID="83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609"
Sep 12 03:05:47.495647 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.495628    2579 scope.go:117] "RemoveContainer" containerID="83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609"
Sep 12 03:05:47.495905 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:05:47.495890    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609\": container with ID starting with 83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609 not found: ID does not exist" containerID="83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609"
Sep 12 03:05:47.495958 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.495913    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609"} err="failed to get container status \"83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609\": rpc error: code = NotFound desc = could not find container \"83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609\": container with ID starting with 83e851f76f966bbdd72169b1de4b8aa99401cadf1ebe780b167c1cc1c769b609 not found: ID does not exist"
Sep 12 03:05:47.507643 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.507615    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"]
Sep 12 03:05:47.510227 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.510205    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d2effd-fdcbcd75-v8rnj"]
Sep 12 03:05:47.576795 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:47.576763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:48.245755 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:48.245727    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed29cc7-8ac8-45dc-a650-409b2ec424dc" path="/var/lib/kubelet/pods/eed29cc7-8ac8-45dc-a650-409b2ec424dc/volumes"
Sep 12 03:05:48.354251 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:48.354219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:49.143139 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:49.143108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:49.598711 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:49.598631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:50.154382 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:50.154353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:50.962038 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:50.962010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:51.645929 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:51.645902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:52.281477 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:52.281433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:53.047028 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:53.046970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:53.656780 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:53.656749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:53.756942 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:53.756913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:53.898304 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:53.898265    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"]
Sep 12 03:05:53.898579 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:53.898555    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy" containerID="cri-o://37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb" gracePeriod=30
Sep 12 03:05:53.920134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:53.920030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:54.582590 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:54.582563    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"]
Sep 12 03:05:54.588826 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:54.588799    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"]
Sep 12 03:05:54.947198 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:54.947167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:05:59.155637 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.155608    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 03:05:59.240901 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.240812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-podinfo\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.240901 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.240870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-data\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.240902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-credential-socket\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.240921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-envoy\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.240948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssjwl\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-kube-api-access-ssjwl\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.240965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-token\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-socket\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istiod-ca-cert\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-certs\") pod \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\" (UID: \"4802f8c1-a1e6-4768-83f4-016ad641e9a9\") "
Sep 12 03:05:59.241424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:59.241424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:59.241424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:59.241683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241492    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:59.241683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241542    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:05:59.241683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241585    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.241683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241604    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.241683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241618    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.241683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.241630    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.243305 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.243283    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:05:59.243408 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.243298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-kube-api-access-ssjwl" (OuterVolumeSpecName: "kube-api-access-ssjwl") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "kube-api-access-ssjwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:59.243408 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.243354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:05:59.243753 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.243734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4802f8c1-a1e6-4768-83f4-016ad641e9a9" (UID: "4802f8c1-a1e6-4768-83f4-016ad641e9a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:05:59.342953 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.342926    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.342953 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.342950    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.342953 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.342959    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.343153 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.342968    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjwl\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-kube-api-access-ssjwl\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.343153 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.342977    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4802f8c1-a1e6-4768-83f4-016ad641e9a9-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:05:59.547901 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.547811    2579 generic.go:356] "Generic (PLEG): container finished" podID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerID="37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb" exitCode=0
Sep 12 03:05:59.547901 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.547862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"
Sep 12 03:05:59.548114 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.547896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" event={"ID":"4802f8c1-a1e6-4768-83f4-016ad641e9a9","Type":"ContainerDied","Data":"37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb"}
Sep 12 03:05:59.548114 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.547948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk" event={"ID":"4802f8c1-a1e6-4768-83f4-016ad641e9a9","Type":"ContainerDied","Data":"5bb623959783a7522808fd2051c9810ca0e4fd08ca75cee998840b285b1589e8"}
Sep 12 03:05:59.548114 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.547970    2579 scope.go:117] "RemoveContainer" containerID="37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb"
Sep 12 03:05:59.557523 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.557501    2579 scope.go:117] "RemoveContainer" containerID="dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"
Sep 12 03:05:59.565766 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.565749    2579 scope.go:117] "RemoveContainer" containerID="37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb"
Sep 12 03:05:59.566037 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:05:59.566017    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb\": container with ID starting with 37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb not found: ID does not exist" containerID="37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb"
Sep 12 03:05:59.566076 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.566048    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb"} err="failed to get container status \"37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb\": rpc error: code = NotFound desc = could not find container \"37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb\": container with ID starting with 37ee01745f43dfe94f19d652740f473240cd5eb5f81fb07ef025717b789febfb not found: ID does not exist"
Sep 12 03:05:59.566076 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.566065    2579 scope.go:117] "RemoveContainer" containerID="dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"
Sep 12 03:05:59.566306 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:05:59.566288    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d\": container with ID starting with dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d not found: ID does not exist" containerID="dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"
Sep 12 03:05:59.566343 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.566314    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d"} err="failed to get container status \"dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d\": rpc error: code = NotFound desc = could not find container \"dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d\": container with ID starting with dbc35da2a4ce7d068612cb4e3e4ab8b6f3af87f2c89b74394bf4f0e4d6d3347d not found: ID does not exist"
Sep 12 03:05:59.569426 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.569406    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"]
Sep 12 03:05:59.574567 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.574545    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d2effd-openshift-default-69b4d8b85d-v7rxk"]
Sep 12 03:05:59.957575 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:05:59.957544    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:00.247648 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:00.247574    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" path="/var/lib/kubelet/pods/4802f8c1-a1e6-4768-83f4-016ad641e9a9/volumes"
Sep 12 03:06:02.543101 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:02.543046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:05.618197 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.618167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:05.639013 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.638983    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"]
Sep 12 03:06:05.639263 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.639240    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w" podUID="1e7fd623-5c80-42df-a7c5-ad74730d720d" containerName="payload-processing" containerID="cri-o://330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56" gracePeriod=30
Sep 12 03:06:05.720831 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.720805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:05.810272 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.810203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:05.870720 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.870629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:05.901751 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:05.901728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 03:06:06.004140 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.004113    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7fd623-5c80-42df-a7c5-ad74730d720d-plugins-config-volume\") pod \"1e7fd623-5c80-42df-a7c5-ad74730d720d\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") "
Sep 12 03:06:06.004317 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.004146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kchq2\" (UniqueName: \"kubernetes.io/projected/1e7fd623-5c80-42df-a7c5-ad74730d720d-kube-api-access-kchq2\") pod \"1e7fd623-5c80-42df-a7c5-ad74730d720d\" (UID: \"1e7fd623-5c80-42df-a7c5-ad74730d720d\") "
Sep 12 03:06:06.004473 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.004434    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7fd623-5c80-42df-a7c5-ad74730d720d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e7fd623-5c80-42df-a7c5-ad74730d720d" (UID: "1e7fd623-5c80-42df-a7c5-ad74730d720d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:06:06.004553 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.004527    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e7fd623-5c80-42df-a7c5-ad74730d720d-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:06.006401 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.006378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7fd623-5c80-42df-a7c5-ad74730d720d-kube-api-access-kchq2" (OuterVolumeSpecName: "kube-api-access-kchq2") pod "1e7fd623-5c80-42df-a7c5-ad74730d720d" (UID: "1e7fd623-5c80-42df-a7c5-ad74730d720d"). InnerVolumeSpecName "kube-api-access-kchq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:06.105912 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.105872    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kchq2\" (UniqueName: \"kubernetes.io/projected/1e7fd623-5c80-42df-a7c5-ad74730d720d-kube-api-access-kchq2\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:06.276189 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.276158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:06.407583 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.407539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:06.583823 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.583737    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e7fd623-5c80-42df-a7c5-ad74730d720d" containerID="330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56" exitCode=0
Sep 12 03:06:06.583823 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.583776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"
Sep 12 03:06:06.584028 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.583819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w" event={"ID":"1e7fd623-5c80-42df-a7c5-ad74730d720d","Type":"ContainerDied","Data":"330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56"}
Sep 12 03:06:06.584028 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.583872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w" event={"ID":"1e7fd623-5c80-42df-a7c5-ad74730d720d","Type":"ContainerDied","Data":"291b70fca3e3e1da15367f614ee583d3b7d2e05873006f76b0b0bc3b24859277"}
Sep 12 03:06:06.584028 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.583896    2579 scope.go:117] "RemoveContainer" containerID="330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56"
Sep 12 03:06:06.592894 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.592873    2579 scope.go:117] "RemoveContainer" containerID="330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56"
Sep 12 03:06:06.593163 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:06:06.593145    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56\": container with ID starting with 330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56 not found: ID does not exist" containerID="330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56"
Sep 12 03:06:06.593214 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.593172    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56"} err="failed to get container status \"330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56\": rpc error: code = NotFound desc = could not find container \"330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56\": container with ID starting with 330c311d15148db29b0ee8941f03b9aca678d75b4485251b8ae2f4012f315b56 not found: ID does not exist"
Sep 12 03:06:06.600379 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.600349    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"]
Sep 12 03:06:06.604396 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.604376    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c6190b-69567c88df-qsx7w"]
Sep 12 03:06:06.874421 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:06.874388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:07.649670 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:07.649642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:08.245733 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:08.245698    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7fd623-5c80-42df-a7c5-ad74730d720d" path="/var/lib/kubelet/pods/1e7fd623-5c80-42df-a7c5-ad74730d720d/volumes"
Sep 12 03:06:09.602994 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.602955    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn" podUID="ab7d4f07-7341-4f89-85c2-8510b9d20f22" containerName="main" containerID="cri-o://bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2" gracePeriod=45
Sep 12 03:06:09.607913 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.607881    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l" podUID="0765346d-1ae2-488a-a0c6-dc5288ffd9bb" containerName="main" containerID="cri-o://6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa" gracePeriod=45
Sep 12 03:06:09.895668 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.895642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:06:09.897717 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.897695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:06:09.941369 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tmp-dir\") pod \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-home\") pod \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941421    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-home\") pod \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wshmf\" (UniqueName: \"kubernetes.io/projected/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-kube-api-access-wshmf\") pod \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0765346d-1ae2-488a-a0c6-dc5288ffd9bb" (UID: "0765346d-1ae2-488a-a0c6-dc5288ffd9bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941606    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tls-certs\") pod \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-model-cache\") pod \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-dshm\") pod \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") "
Sep 12 03:06:09.941738 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tls-certs\") pod \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") "
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tpsg\" (UniqueName: \"kubernetes.io/projected/ab7d4f07-7341-4f89-85c2-8510b9d20f22-kube-api-access-4tpsg\") pod \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") "
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-model-cache\") pod \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") "
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-dshm\") pod \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\" (UID: \"0765346d-1ae2-488a-a0c6-dc5288ffd9bb\") "
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tmp-dir\") pod \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\" (UID: \"ab7d4f07-7341-4f89-85c2-8510b9d20f22\") "
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-home" (OuterVolumeSpecName: "home") pod "0765346d-1ae2-488a-a0c6-dc5288ffd9bb" (UID: "0765346d-1ae2-488a-a0c6-dc5288ffd9bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.941750    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-home" (OuterVolumeSpecName: "home") pod "ab7d4f07-7341-4f89-85c2-8510b9d20f22" (UID: "ab7d4f07-7341-4f89-85c2-8510b9d20f22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.942125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab7d4f07-7341-4f89-85c2-8510b9d20f22" (UID: "ab7d4f07-7341-4f89-85c2-8510b9d20f22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.942201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.942185    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:09.942691 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.942410    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "0765346d-1ae2-488a-a0c6-dc5288ffd9bb" (UID: "0765346d-1ae2-488a-a0c6-dc5288ffd9bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.942691 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.942416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab7d4f07-7341-4f89-85c2-8510b9d20f22" (UID: "ab7d4f07-7341-4f89-85c2-8510b9d20f22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.945130 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.945094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-dshm" (OuterVolumeSpecName: "dshm") pod "ab7d4f07-7341-4f89-85c2-8510b9d20f22" (UID: "ab7d4f07-7341-4f89-85c2-8510b9d20f22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.945130 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.945101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7d4f07-7341-4f89-85c2-8510b9d20f22-kube-api-access-4tpsg" (OuterVolumeSpecName: "kube-api-access-4tpsg") pod "ab7d4f07-7341-4f89-85c2-8510b9d20f22" (UID: "ab7d4f07-7341-4f89-85c2-8510b9d20f22"). InnerVolumeSpecName "kube-api-access-4tpsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:09.945329 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.945143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-kube-api-access-wshmf" (OuterVolumeSpecName: "kube-api-access-wshmf") pod "0765346d-1ae2-488a-a0c6-dc5288ffd9bb" (UID: "0765346d-1ae2-488a-a0c6-dc5288ffd9bb"). InnerVolumeSpecName "kube-api-access-wshmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:06:09.945329 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.945289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-dshm" (OuterVolumeSpecName: "dshm") pod "0765346d-1ae2-488a-a0c6-dc5288ffd9bb" (UID: "0765346d-1ae2-488a-a0c6-dc5288ffd9bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:06:09.945538 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.945520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab7d4f07-7341-4f89-85c2-8510b9d20f22" (UID: "ab7d4f07-7341-4f89-85c2-8510b9d20f22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:09.945914 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:09.945898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0765346d-1ae2-488a-a0c6-dc5288ffd9bb" (UID: "0765346d-1ae2-488a-a0c6-dc5288ffd9bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:06:10.043422 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043389    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043422 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043414    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043422 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043425    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wshmf\" (UniqueName: \"kubernetes.io/projected/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-kube-api-access-wshmf\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043435    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043467    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043474    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043481    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpsg\" (UniqueName: \"kubernetes.io/projected/ab7d4f07-7341-4f89-85c2-8510b9d20f22-kube-api-access-4tpsg\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043496    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043503    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0765346d-1ae2-488a-a0c6-dc5288ffd9bb-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.043663 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.043510    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7d4f07-7341-4f89-85c2-8510b9d20f22-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:06:10.605980 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.605948    2579 generic.go:356] "Generic (PLEG): container finished" podID="0765346d-1ae2-488a-a0c6-dc5288ffd9bb" containerID="6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa" exitCode=0
Sep 12 03:06:10.606494 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.605993    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"
Sep 12 03:06:10.606494 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.606040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l" event={"ID":"0765346d-1ae2-488a-a0c6-dc5288ffd9bb","Type":"ContainerDied","Data":"6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa"}
Sep 12 03:06:10.606494 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.606089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l" event={"ID":"0765346d-1ae2-488a-a0c6-dc5288ffd9bb","Type":"ContainerDied","Data":"a4b270fc6f16ff6ac783c60df6d874c257890cd9c11abe3b09d9a4aa00820240"}
Sep 12 03:06:10.606494 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.606108    2579 scope.go:117] "RemoveContainer" containerID="6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa"
Sep 12 03:06:10.607599 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.607577    2579 generic.go:356] "Generic (PLEG): container finished" podID="ab7d4f07-7341-4f89-85c2-8510b9d20f22" containerID="bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2" exitCode=0
Sep 12 03:06:10.607688 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.607627    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"
Sep 12 03:06:10.607688 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.607659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn" event={"ID":"ab7d4f07-7341-4f89-85c2-8510b9d20f22","Type":"ContainerDied","Data":"bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2"}
Sep 12 03:06:10.607804 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.607697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn" event={"ID":"ab7d4f07-7341-4f89-85c2-8510b9d20f22","Type":"ContainerDied","Data":"9d80112be636593d08ae24c8879e01a5fe0cfb5df3464fe145d7d4c42108db49"}
Sep 12 03:06:10.616354 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.616331    2579 scope.go:117] "RemoveContainer" containerID="6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa"
Sep 12 03:06:10.616631 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:06:10.616612    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa\": container with ID starting with 6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa not found: ID does not exist" containerID="6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa"
Sep 12 03:06:10.616708 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.616639    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa"} err="failed to get container status \"6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa\": rpc error: code = NotFound desc = could not find container \"6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa\": container with ID starting with 6065b5e87651e48e7f30ed30435f48b0bc318b4abdbcba809173a869d7e17afa not found: ID does not exist"
Sep 12 03:06:10.616708 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.616655    2579 scope.go:117] "RemoveContainer" containerID="bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2"
Sep 12 03:06:10.622117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.622101    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"]
Sep 12 03:06:10.625056 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.625038    2579 scope.go:117] "RemoveContainer" containerID="bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2"
Sep 12 03:06:10.625327 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:06:10.625309    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2\": container with ID starting with bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2 not found: ID does not exist" containerID="bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2"
Sep 12 03:06:10.625393 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.625338    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2"} err="failed to get container status \"bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2\": rpc error: code = NotFound desc = could not find container \"bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2\": container with ID starting with bc3f9b0a9f517a417b0afc5bc1cc46d4990c502b4075e275a72a070a81c293c2 not found: ID does not exist"
Sep 12 03:06:10.625740 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.625723    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-w4-c6190b-kserve-6748b7fdc4-7ns7l"]
Sep 12 03:06:10.640338 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.640317    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"]
Sep 12 03:06:10.644243 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:10.644225    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c6190b/distinct-2-w4-c6190b-kserve-5d7cdb99c7-chbpn"]
Sep 12 03:06:12.247000 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:12.246964    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0765346d-1ae2-488a-a0c6-dc5288ffd9bb" path="/var/lib/kubelet/pods/0765346d-1ae2-488a-a0c6-dc5288ffd9bb/volumes"
Sep 12 03:06:12.247369 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:12.247351    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7d4f07-7341-4f89-85c2-8510b9d20f22" path="/var/lib/kubelet/pods/ab7d4f07-7341-4f89-85c2-8510b9d20f22/volumes"
Sep 12 03:06:12.372765 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:12.372731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:12.473845 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:12.473819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:12.640345 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:12.640319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:19.614031 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:19.614001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:22.274927 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:22.274899    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:06:22.278303 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:22.278285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:06:22.910904 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:22.910874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:31.486555 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:31.486521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:31.586693 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:31.586662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:31.761091 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:31.761001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:34.200120 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200081    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"]
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200435    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200465    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200476    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200481    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200488    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c90f719-277e-4f8a-b672-046d026cdd3c" containerName="main"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200493    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c90f719-277e-4f8a-b672-046d026cdd3c" containerName="main"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200499    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy"
Sep 12 03:06:34.200505 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200507    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200514    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200519    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200526    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200533    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200546    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0765346d-1ae2-488a-a0c6-dc5288ffd9bb" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200551    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0765346d-1ae2-488a-a0c6-dc5288ffd9bb" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200559    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed29cc7-8ac8-45dc-a650-409b2ec424dc" containerName="maas-api"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed29cc7-8ac8-45dc-a650-409b2ec424dc" containerName="maas-api"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200569    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067a47dc-2c59-4898-bf89-5b8063f5f90a" containerName="maas-api"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200574    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="067a47dc-2c59-4898-bf89-5b8063f5f90a" containerName="maas-api"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200580    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ec080c-43b7-4907-b985-b8deaa6ce6fd" containerName="payload-pre-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200584    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ec080c-43b7-4907-b985-b8deaa6ce6fd" containerName="payload-pre-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200592    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7fd623-5c80-42df-a7c5-ad74730d720d" containerName="payload-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200596    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7fd623-5c80-42df-a7c5-ad74730d720d" containerName="payload-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200602    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de4c729-3351-428c-8cc3-78ed7cd2c751" containerName="payload-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200608    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de4c729-3351-428c-8cc3-78ed7cd2c751" containerName="payload-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200618    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7d4f07-7341-4f89-85c2-8510b9d20f22" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200623    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7d4f07-7341-4f89-85c2-8510b9d20f22" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200677    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200686    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200696    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0765346d-1ae2-488a-a0c6-dc5288ffd9bb" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200737    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de4c729-3351-428c-8cc3-78ed7cd2c751" containerName="payload-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200748    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee7e2bb2-01fe-41bb-a3e2-04f25ea53431" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200757    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200766    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ec080c-43b7-4907-b985-b8deaa6ce6fd" containerName="payload-pre-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200776    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200806    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7d4f07-7341-4f89-85c2-8510b9d20f22" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200817    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c90f719-277e-4f8a-b672-046d026cdd3c" containerName="main"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200827    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed29cc7-8ac8-45dc-a650-409b2ec424dc" containerName="maas-api"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200836    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7fd623-5c80-42df-a7c5-ad74730d720d" containerName="payload-processing"
Sep 12 03:06:34.200964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.200845    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="067a47dc-2c59-4898-bf89-5b8063f5f90a" containerName="maas-api"
Sep 12 03:06:34.203932 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.203917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.207686 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.207663    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"openshift-service-ca.crt\""
Sep 12 03:06:34.207805 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.207768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"default-dockercfg-pc6s2\""
Sep 12 03:06:34.207805 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.207768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"sub-test-model-f91169-kserve-self-signed-certs\""
Sep 12 03:06:34.207939 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.207928    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f91169\"/\"kube-root-ca.crt\""
Sep 12 03:06:34.214792 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.214766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"]
Sep 12 03:06:34.256639 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.256610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-dshm\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.256780 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.256643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjjz\" (UniqueName: \"kubernetes.io/projected/1475916f-301f-4901-88fb-07756cf7278c-kube-api-access-nbjjz\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.256780 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.256678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1475916f-301f-4901-88fb-07756cf7278c-tls-certs\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.256780 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.256721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-tmp-dir\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.256780 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.256764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-model-cache\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.256952 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.256785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-home\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.315315 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.315283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:34.357213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1475916f-301f-4901-88fb-07756cf7278c-tls-certs\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357372 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-tmp-dir\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357372 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-model-cache\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357372 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-home\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357556 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-dshm\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357556 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjjz\" (UniqueName: \"kubernetes.io/projected/1475916f-301f-4901-88fb-07756cf7278c-kube-api-access-nbjjz\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357668 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-tmp-dir\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357741 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-model-cache\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.357811 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.357794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-home\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.359702 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.359685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-dshm\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.359776 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.359755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1475916f-301f-4901-88fb-07756cf7278c-tls-certs\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.365067 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.365042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjjz\" (UniqueName: \"kubernetes.io/projected/1475916f-301f-4901-88fb-07756cf7278c-kube-api-access-nbjjz\") pod \"sub-test-model-f91169-kserve-598b9f9b7f-fbqcm\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") " pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.514015 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.513926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:34.783727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:34.783647    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"]
Sep 12 03:06:34.787081 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:06:34.787055    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1475916f_301f_4901_88fb_07756cf7278c.slice/crio-0048e2b8d563056b44654bd3f9f6853615885ebed34cd0e499162a122d8b5ae7 WatchSource:0}: Error finding container 0048e2b8d563056b44654bd3f9f6853615885ebed34cd0e499162a122d8b5ae7: Status 404 returned error can't find the container with id 0048e2b8d563056b44654bd3f9f6853615885ebed34cd0e499162a122d8b5ae7
Sep 12 03:06:35.730174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:35.730135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm" event={"ID":"1475916f-301f-4901-88fb-07756cf7278c","Type":"ContainerStarted","Data":"ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752"}
Sep 12 03:06:35.730174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:35.730178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm" event={"ID":"1475916f-301f-4901-88fb-07756cf7278c","Type":"ContainerStarted","Data":"0048e2b8d563056b44654bd3f9f6853615885ebed34cd0e499162a122d8b5ae7"}
Sep 12 03:06:35.746063 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:35.746017    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm" podStartSLOduration=1.746003125 podStartE2EDuration="1.746003125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:06:35.745645592 +0000 UTC m=+1814.028559174" watchObservedRunningTime="2026-09-12 03:06:35.746003125 +0000 UTC m=+1814.028916705"
Sep 12 03:06:37.558234 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:37.558201    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:38.291470 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:38.291416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:43.129777 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:43.129744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:06:44.514897 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:44.514855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:44.515298 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:44.514910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:44.527218 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:44.527185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:06:44.782262 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:06:44.782185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:07:10.513110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:07:10.513073    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:07:48.397858 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:07:48.397764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"]
Sep 12 03:07:48.418494 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:07:48.418441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:07:49.416438 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:07:49.416405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:08:03.157782 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.157755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:08:03.417821 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.417714    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm" podUID="1475916f-301f-4901-88fb-07756cf7278c" containerName="main" containerID="cri-o://ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752" gracePeriod=45
Sep 12 03:08:03.669716 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.669655    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:08:03.826664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1475916f-301f-4901-88fb-07756cf7278c-tls-certs\") pod \"1475916f-301f-4901-88fb-07756cf7278c\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") "
Sep 12 03:08:03.826844 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-home\") pod \"1475916f-301f-4901-88fb-07756cf7278c\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") "
Sep 12 03:08:03.826844 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-tmp-dir\") pod \"1475916f-301f-4901-88fb-07756cf7278c\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") "
Sep 12 03:08:03.826844 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-dshm\") pod \"1475916f-301f-4901-88fb-07756cf7278c\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") "
Sep 12 03:08:03.826844 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-model-cache\") pod \"1475916f-301f-4901-88fb-07756cf7278c\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") "
Sep 12 03:08:03.827055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbjjz\" (UniqueName: \"kubernetes.io/projected/1475916f-301f-4901-88fb-07756cf7278c-kube-api-access-nbjjz\") pod \"1475916f-301f-4901-88fb-07756cf7278c\" (UID: \"1475916f-301f-4901-88fb-07756cf7278c\") "
Sep 12 03:08:03.827055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-home" (OuterVolumeSpecName: "home") pod "1475916f-301f-4901-88fb-07756cf7278c" (UID: "1475916f-301f-4901-88fb-07756cf7278c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:03.827055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.826989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1475916f-301f-4901-88fb-07756cf7278c" (UID: "1475916f-301f-4901-88fb-07756cf7278c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:03.827214 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.827132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1475916f-301f-4901-88fb-07756cf7278c" (UID: "1475916f-301f-4901-88fb-07756cf7278c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:03.827273 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.827223    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:08:03.827273 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.827245    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:08:03.827273 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.827258    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:08:03.829156 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.829136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1475916f-301f-4901-88fb-07756cf7278c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1475916f-301f-4901-88fb-07756cf7278c" (UID: "1475916f-301f-4901-88fb-07756cf7278c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:08:03.829254 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.829172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-dshm" (OuterVolumeSpecName: "dshm") pod "1475916f-301f-4901-88fb-07756cf7278c" (UID: "1475916f-301f-4901-88fb-07756cf7278c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:08:03.829254 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.829235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1475916f-301f-4901-88fb-07756cf7278c-kube-api-access-nbjjz" (OuterVolumeSpecName: "kube-api-access-nbjjz") pod "1475916f-301f-4901-88fb-07756cf7278c" (UID: "1475916f-301f-4901-88fb-07756cf7278c"). InnerVolumeSpecName "kube-api-access-nbjjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:08:03.928330 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.928245    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1475916f-301f-4901-88fb-07756cf7278c-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:08:03.928330 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.928274    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjjz\" (UniqueName: \"kubernetes.io/projected/1475916f-301f-4901-88fb-07756cf7278c-kube-api-access-nbjjz\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:08:03.928330 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:03.928289    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1475916f-301f-4901-88fb-07756cf7278c-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:08:04.144211 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.144177    2579 generic.go:356] "Generic (PLEG): container finished" podID="1475916f-301f-4901-88fb-07756cf7278c" containerID="ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752" exitCode=0
Sep 12 03:08:04.144375 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.144220    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"
Sep 12 03:08:04.144375 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.144247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm" event={"ID":"1475916f-301f-4901-88fb-07756cf7278c","Type":"ContainerDied","Data":"ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752"}
Sep 12 03:08:04.144375 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.144279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm" event={"ID":"1475916f-301f-4901-88fb-07756cf7278c","Type":"ContainerDied","Data":"0048e2b8d563056b44654bd3f9f6853615885ebed34cd0e499162a122d8b5ae7"}
Sep 12 03:08:04.144375 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.144299    2579 scope.go:117] "RemoveContainer" containerID="ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752"
Sep 12 03:08:04.153709 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.153693    2579 scope.go:117] "RemoveContainer" containerID="ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752"
Sep 12 03:08:04.153988 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:08:04.153970    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752\": container with ID starting with ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752 not found: ID does not exist" containerID="ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752"
Sep 12 03:08:04.154055 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.154000    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752"} err="failed to get container status \"ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752\": rpc error: code = NotFound desc = could not find container \"ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752\": container with ID starting with ff974f547c278255fa650c22134a1cffc37d1f785a500f271175e176df3a3752 not found: ID does not exist"
Sep 12 03:08:04.166746 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.166716    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"]
Sep 12 03:08:04.167830 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.167810    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f91169/sub-test-model-f91169-kserve-598b9f9b7f-fbqcm"]
Sep 12 03:08:04.245902 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:04.245827    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1475916f-301f-4901-88fb-07756cf7278c" path="/var/lib/kubelet/pods/1475916f-301f-4901-88fb-07756cf7278c/volumes"
Sep 12 03:08:48.631981 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:08:48.631943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:26.116645 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:26.116560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:27.050314 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.050255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:27.519483 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.519433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:27.820622 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.820531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"]
Sep 12 03:09:27.820996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.820975    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy"
Sep 12 03:09:27.820996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.820996    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4802f8c1-a1e6-4768-83f4-016ad641e9a9" containerName="istio-proxy"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821008    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821017    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821035    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1475916f-301f-4901-88fb-07756cf7278c" containerName="main"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821043    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1475916f-301f-4901-88fb-07756cf7278c" containerName="main"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821156    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1475916f-301f-4901-88fb-07756cf7278c" containerName="main"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821172    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="574e6656-2c15-44ef-8f77-27a10fc503b3" containerName="istio-proxy"
Sep 12 03:09:27.821202 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.821183    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5b6443-0d70-4ab5-84b1-ec1a12987d65" containerName="istio-proxy"
Sep 12 03:09:27.824536 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.824509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.827172 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.827149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ad5a1b-openshift-default-dockercfg-2l5hq\""
Sep 12 03:09:27.835490 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.835470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"]
Sep 12 03:09:27.848698 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.848676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:27.954690 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-podinfo\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.954690 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istiod-ca-cert\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.954937 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-token\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.954937 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-certs\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.954937 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-data\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.954937 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-socket\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.955070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-credential-socket\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.955070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc482\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-kube-api-access-xc482\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:27.955070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:27.954999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-envoy\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056232 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-certs\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056232 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-data\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-socket\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-credential-socket\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc482\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-kube-api-access-xc482\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-envoy\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-podinfo\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istiod-ca-cert\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-token\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056919 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-data\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056919 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-credential-socket\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.056919 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.056841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-certs\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.057196 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.057174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-socket\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.057349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.057332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istiod-ca-cert\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.058904 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.058883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-envoy\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.059174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.059157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-podinfo\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.065020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.064992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-token\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.065579 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.065560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc482\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-kube-api-access-xc482\") pod \"e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") " pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.135789 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.135760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:28.376329 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.376296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:28.413419 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.413386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"]
Sep 12 03:09:28.415493 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:09:28.415465    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c13dd_ffcf_407a_9f61_6c67daa22e3b.slice/crio-08e00225d2c33cc0cfedbb3304a5cd7767ad5281fbf8c0fddc0b452efb886216 WatchSource:0}: Error finding container 08e00225d2c33cc0cfedbb3304a5cd7767ad5281fbf8c0fddc0b452efb886216: Status 404 returned error can't find the container with id 08e00225d2c33cc0cfedbb3304a5cd7767ad5281fbf8c0fddc0b452efb886216
Sep 12 03:09:28.417784 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.417745    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:28.417876 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.417831    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:28.417918 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.417880    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:28.535545 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.535507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" event={"ID":"719c13dd-ffcf-407a-9f61-6c67daa22e3b","Type":"ContainerStarted","Data":"1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"}
Sep 12 03:09:28.535958 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.535577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" event={"ID":"719c13dd-ffcf-407a-9f61-6c67daa22e3b","Type":"ContainerStarted","Data":"08e00225d2c33cc0cfedbb3304a5cd7767ad5281fbf8c0fddc0b452efb886216"}
Sep 12 03:09:28.554596 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:28.554530    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podStartSLOduration=1.5545110549999999 podStartE2EDuration="1.554511055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:09:28.552952024 +0000 UTC m=+1986.835865606" watchObservedRunningTime="2026-09-12 03:09:28.554511055 +0000 UTC m=+1986.837424692"
Sep 12 03:09:29.136499 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:29.136463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:29.141880 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:29.141858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:29.540364 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:29.540288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:29.541341 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:29.541322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:30.357215 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:30.357163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:30.528172 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:30.528140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:30.627540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:30.627458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:31.146241 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.146210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:31.913178 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.913136    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"]
Sep 12 03:09:31.917193 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.917159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:31.920395 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.920301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ad5a1b-serving-cert\""
Sep 12 03:09:31.929007 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.928978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"]
Sep 12 03:09:31.946599 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.946559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:31.995013 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.994979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:31.995013 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.995014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rbt\" (UniqueName: \"kubernetes.io/projected/595fdf88-87c7-42dc-b860-4e4d6210d69a-kube-api-access-s9rbt\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:31.995230 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.995041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:31.995230 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:31.995177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595fdf88-87c7-42dc-b860-4e4d6210d69a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.007062 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.007028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:32.096662 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.096625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.096853 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.096665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rbt\" (UniqueName: \"kubernetes.io/projected/595fdf88-87c7-42dc-b860-4e4d6210d69a-kube-api-access-s9rbt\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.096853 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.096697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.096853 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.096768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595fdf88-87c7-42dc-b860-4e4d6210d69a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.097016 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:09:32.096875    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-serving-cert: secret "maas-api-e2e-ipp-a-ad5a1b-serving-cert" not found
Sep 12 03:09:32.097016 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:09:32.096961    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls podName:595fdf88-87c7-42dc-b860-4e4d6210d69a nodeName:}" failed. No retries permitted until 2026-09-12 03:09:32.596939296 +0000 UTC m=+1990.879852855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls") pod "maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" (UID: "595fdf88-87c7-42dc-b860-4e4d6210d69a") : secret "maas-api-e2e-ipp-a-ad5a1b-serving-cert" not found
Sep 12 03:09:32.097532 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.097512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595fdf88-87c7-42dc-b860-4e4d6210d69a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.099708 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.099679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.105739 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.105711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rbt\" (UniqueName: \"kubernetes.io/projected/595fdf88-87c7-42dc-b860-4e4d6210d69a-kube-api-access-s9rbt\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.204187 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.204095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:32.432244 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.432214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:32.530352 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.530256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:32.601792 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.601748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.604316 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.604296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:32.829627 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:32.829536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:33.105539 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:33.105513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"]
Sep 12 03:09:33.108325 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:09:33.108299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595fdf88_87c7_42dc_b860_4e4d6210d69a.slice/crio-2992e8ef3a0a7813d8a347c167cc5cac36b13d9d4829d1a4443a770dc90f5f7b WatchSource:0}: Error finding container 2992e8ef3a0a7813d8a347c167cc5cac36b13d9d4829d1a4443a770dc90f5f7b: Status 404 returned error can't find the container with id 2992e8ef3a0a7813d8a347c167cc5cac36b13d9d4829d1a4443a770dc90f5f7b
Sep 12 03:09:33.109897 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:33.109878    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 03:09:33.564462 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:33.564359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" event={"ID":"595fdf88-87c7-42dc-b860-4e4d6210d69a","Type":"ContainerStarted","Data":"2992e8ef3a0a7813d8a347c167cc5cac36b13d9d4829d1a4443a770dc90f5f7b"}
Sep 12 03:09:34.570797 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:34.570764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" event={"ID":"595fdf88-87c7-42dc-b860-4e4d6210d69a","Type":"ContainerStarted","Data":"4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e"}
Sep 12 03:09:34.571197 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:34.570895    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:34.590232 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:34.590181    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" podStartSLOduration=3.103379212 podStartE2EDuration="3.590165242s" totalImagesPullingTime="486.78603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:09:34.588404102 +0000 UTC m=+1992.871317684" watchObservedRunningTime="2026-09-12 03:09:34.590165242 +0000 UTC m=+1992.873078846"
Sep 12 03:09:40.587466 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:40.587427    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:09:45.582573 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:45.582530    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 03:09:45.582964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:45.582581    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 03:09:47.635720 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:47.635687    2579 generic.go:356] "Generic (PLEG): container finished" podID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerID="1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4" exitCode=0
Sep 12 03:09:47.636135 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:47.635726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" event={"ID":"719c13dd-ffcf-407a-9f61-6c67daa22e3b","Type":"ContainerDied","Data":"1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"}
Sep 12 03:09:47.636183 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:47.636173    2579 scope.go:117] "RemoveContainer" containerID="1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"
Sep 12 03:09:47.637326 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:47.637300    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:47.637395 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:47.637363    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:47.637437 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:47.637395    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:48.643503 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:48.643467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" event={"ID":"719c13dd-ffcf-407a-9f61-6c67daa22e3b","Type":"ContainerStarted","Data":"f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d"}
Sep 12 03:09:49.136939 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:49.136902    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:49.137174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:49.137150    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 03:09:49.137261 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:49.137205    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 03:09:50.136686 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:50.136648    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 03:09:50.137071 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:50.136716    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 03:09:51.136349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:51.136313    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 03:09:51.136560 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:51.136383    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 03:09:52.136886 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:52.136840    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 03:09:52.137279 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:52.136902    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 03:09:53.136164 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:53.136120    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 03:09:53.136325 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:53.136192    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 03:09:54.136070 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:54.136034    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 03:09:54.136489 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:54.136090    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 03:09:54.145038 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:54.145007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:54.848613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:54.848566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:55.139616 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:55.139587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:55.140012 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:55.139937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:55.140711 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:55.140694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:09:55.609320 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:55.609266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:55.710288 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:55.710254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:56.331627 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.331594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:56.433465 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.433422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:56.616114 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.616076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"]
Sep 12 03:09:56.618896 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.618875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.621966 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.621938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-27bf83-openshift-default-dockercfg-q4qw8\""
Sep 12 03:09:56.630603 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.630583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"]
Sep 12 03:09:56.637408 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-credential-socket\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637523 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-socket\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637523 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-token\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637523 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-certs\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637523 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8afb8f-8681-4466-99cb-0229d84deacd-istiod-ca-cert\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637667 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nkt\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-kube-api-access-s7nkt\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637667 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-envoy\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637667 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-podinfo\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.637667 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.637648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-data\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.643118 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.643096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:56.738536 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-data\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.738743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-credential-socket\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.738743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-socket\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.738743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-token\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.738743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-certs\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.738743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8afb8f-8681-4466-99cb-0229d84deacd-istiod-ca-cert\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nkt\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-kube-api-access-s7nkt\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-envoy\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-podinfo\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.738973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-data\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.739005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-credential-socket\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739250 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.739090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-socket\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739250 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.739222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-certs\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.739552 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.739529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8afb8f-8681-4466-99cb-0229d84deacd-istiod-ca-cert\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.741220 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.741201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-envoy\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.741605 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.741585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-podinfo\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.749681 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.749650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:56.750102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.750074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nkt\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-kube-api-access-s7nkt\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.750338 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.750317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-token\") pod \"e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") " pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:56.931478 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:56.931375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:57.207965 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:09:57.207934    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8afb8f_8681_4466_99cb_0229d84deacd.slice/crio-0d112e9cc8050974aa86e2c1c42fa75962f65dfe93afa6f261f5faade4a3d4c6 WatchSource:0}: Error finding container 0d112e9cc8050974aa86e2c1c42fa75962f65dfe93afa6f261f5faade4a3d4c6: Status 404 returned error can't find the container with id 0d112e9cc8050974aa86e2c1c42fa75962f65dfe93afa6f261f5faade4a3d4c6
Sep 12 03:09:57.210108 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.210079    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"]
Sep 12 03:09:57.210478 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.210429    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:57.210545 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.210514    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:57.210545 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.210541    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:09:57.663917 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.663875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:57.693165 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.693131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" event={"ID":"fc8afb8f-8681-4466-99cb-0229d84deacd","Type":"ContainerStarted","Data":"a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"}
Sep 12 03:09:57.693343 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.693181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" event={"ID":"fc8afb8f-8681-4466-99cb-0229d84deacd","Type":"ContainerStarted","Data":"0d112e9cc8050974aa86e2c1c42fa75962f65dfe93afa6f261f5faade4a3d4c6"}
Sep 12 03:09:57.714879 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.714826    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podStartSLOduration=1.71480641 podStartE2EDuration="1.71480641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:09:57.71363477 +0000 UTC m=+2015.996548367" watchObservedRunningTime="2026-09-12 03:09:57.71480641 +0000 UTC m=+2015.997719992"
Sep 12 03:09:57.931577 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:57.931498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:58.934720 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:58.934676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:58.937095 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:58.937072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:59.558131 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:59.558099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:59.663009 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:59.662971    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:09:59.705640 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:59.705606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:59.706697 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:59.706675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:09:59.731562 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:09:59.731537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:00.473815 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:00.473767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:01.257653 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:01.257616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:02.051390 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.051336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:02.148564 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.148536    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:02.514657 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.514623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:02.610623 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.610587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:02.641031 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.641004    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"]
Sep 12 03:10:02.643635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.643615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.646792 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.646768    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-27bf83\""
Sep 12 03:10:02.646906 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.646810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-27bf83-dockercfg-rzqqj\""
Sep 12 03:10:02.654391 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.654370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"]
Sep 12 03:10:02.689240 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.689216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhcp\" (UniqueName: \"kubernetes.io/projected/15b3291a-b81d-4795-b464-d1b729ea2e56-kube-api-access-zbhcp\") pod \"payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.689410 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.689271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b3291a-b81d-4795-b464-d1b729ea2e56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.790648 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.790562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhcp\" (UniqueName: \"kubernetes.io/projected/15b3291a-b81d-4795-b464-d1b729ea2e56-kube-api-access-zbhcp\") pod \"payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.790800 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.790654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b3291a-b81d-4795-b464-d1b729ea2e56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.791556 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.791534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b3291a-b81d-4795-b464-d1b729ea2e56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.799217 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.799184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhcp\" (UniqueName: \"kubernetes.io/projected/15b3291a-b81d-4795-b464-d1b729ea2e56-kube-api-access-zbhcp\") pod \"payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:02.956312 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:02.956272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:03.230613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:03.229939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"]
Sep 12 03:10:03.232922 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:10:03.232894    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b3291a_b81d_4795_b464_d1b729ea2e56.slice/crio-8296564e62ce21f2784c5ffb460c2e91107a2381bb2ea8bb84673ecedcec7be5 WatchSource:0}: Error finding container 8296564e62ce21f2784c5ffb460c2e91107a2381bb2ea8bb84673ecedcec7be5: Status 404 returned error can't find the container with id 8296564e62ce21f2784c5ffb460c2e91107a2381bb2ea8bb84673ecedcec7be5
Sep 12 03:10:03.729889 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:03.729851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x" event={"ID":"15b3291a-b81d-4795-b464-d1b729ea2e56","Type":"ContainerStarted","Data":"e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5"}
Sep 12 03:10:03.730092 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:03.729902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x" event={"ID":"15b3291a-b81d-4795-b464-d1b729ea2e56","Type":"ContainerStarted","Data":"8296564e62ce21f2784c5ffb460c2e91107a2381bb2ea8bb84673ecedcec7be5"}
Sep 12 03:10:03.730092 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:03.729934    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:03.749136 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:03.749080    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x" podStartSLOduration=1.749056633 podStartE2EDuration="1.749056633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:10:03.745926638 +0000 UTC m=+2022.028840219" watchObservedRunningTime="2026-09-12 03:10:03.749056633 +0000 UTC m=+2022.031970218"
Sep 12 03:10:14.736621 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:14.736593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:10:18.814386 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:18.814352    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerID="a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348" exitCode=0
Sep 12 03:10:18.814837 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:18.814428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" event={"ID":"fc8afb8f-8681-4466-99cb-0229d84deacd","Type":"ContainerDied","Data":"a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"}
Sep 12 03:10:18.814837 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:18.814811    2579 scope.go:117] "RemoveContainer" containerID="a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"
Sep 12 03:10:18.815880 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:18.815848    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:10:18.815934 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:18.815918    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:10:18.815970 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:18.815949    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:10:19.821782 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:19.821749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" event={"ID":"fc8afb8f-8681-4466-99cb-0229d84deacd","Type":"ContainerStarted","Data":"4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a"}
Sep 12 03:10:19.932199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:19.932161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:10:19.932513 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:19.932488    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 12 03:10:19.932635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:19.932550    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 12 03:10:20.932102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:20.932068    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 12 03:10:20.932550 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:20.932135    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 12 03:10:21.932565 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:21.932527    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 12 03:10:21.932969 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:21.932581    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 12 03:10:22.438721 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:22.438678    2579 scope.go:117] "RemoveContainer" containerID="b3bf5b5c2a2a47081f9ab436ecd913d4c472da10e289fb375d7481fd4ad804f5"
Sep 12 03:10:22.932086 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:22.932049    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 12 03:10:22.932269 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:22.932105    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 12 03:10:23.932211 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:23.932159    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 12 03:10:23.932635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:23.932230    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 12 03:10:24.719285 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:24.719245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:24.943727 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:24.943685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:24.985958 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:24.985868    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:10:24.986269 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:24.986245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:10:24.987110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:24.987095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:10:25.753265 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:25.753231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:26.433763 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:26.433724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:27.212659 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:27.212625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:27.870117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:27.870080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:39.984258 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:39.984229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l7qnf_0b574436-4216-4db3-8a6d-f53d83910cf7/payload-processing/0.log"
Sep 12 03:10:40.109358 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:40.109326    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x_15b3291a-b81d-4795-b464-d1b729ea2e56/payload-processing/0.log"
Sep 12 03:10:44.554869 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:44.554788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:48.852876 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:48.852842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:10:49.953218 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:10:49.953179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:03.337108 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:03.337075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:11.099839 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.099803    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l7qnf_0b574436-4216-4db3-8a6d-f53d83910cf7/payload-processing/0.log"
Sep 12 03:11:11.221657 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.221631    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x_15b3291a-b81d-4795-b464-d1b729ea2e56/payload-processing/0.log"
Sep 12 03:11:11.505397 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.505368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:11.810502 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.809510    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"]
Sep 12 03:11:11.814990 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.814964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.817652 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.817629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-78f530-openshift-default-dockercfg-84skg\""
Sep 12 03:11:11.828393 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.828370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"]
Sep 12 03:11:11.836475 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.836426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:11.853523 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-data\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-socket\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037a1dc0-54c7-460b-9968-f92128742f37-istiod-ca-cert\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853747 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-credential-socket\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853747 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-istio-token\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853747 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-kube-api-access-l94dj\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853747 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-certs\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853994 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-envoy\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.853994 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.853852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037a1dc0-54c7-460b-9968-f92128742f37-istio-podinfo\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.932596 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.932525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:11.954996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.954876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-kube-api-access-l94dj\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.954996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.954931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-certs\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.954996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.954963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-envoy\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.955644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037a1dc0-54c7-460b-9968-f92128742f37-istio-podinfo\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.955644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-data\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.955644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-socket\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.955644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037a1dc0-54c7-460b-9968-f92128742f37-istiod-ca-cert\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.955644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-credential-socket\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.955644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-istio-token\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.956178 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-certs\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.956178 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.955939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-socket\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.956291 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.956253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-credential-socket\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.956472 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.956411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037a1dc0-54c7-460b-9968-f92128742f37-istiod-ca-cert\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.956644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.956518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-data\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.957760 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.957738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-envoy\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.958558 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.958539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037a1dc0-54c7-460b-9968-f92128742f37-istio-podinfo\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.963907 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.963888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-istio-token\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:11.964095 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:11.964078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-kube-api-access-l94dj\") pod \"e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") " pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:12.124830 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:12.124811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:12.416893 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:12.416858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"]
Sep 12 03:11:12.418464 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:11:12.418418    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037a1dc0_54c7_460b_9968_f92128742f37.slice/crio-63c235d03f9866239e59deeebf60cbd12b3c92ae40e3ad9c6f2546a839bc5628 WatchSource:0}: Error finding container 63c235d03f9866239e59deeebf60cbd12b3c92ae40e3ad9c6f2546a839bc5628: Status 404 returned error can't find the container with id 63c235d03f9866239e59deeebf60cbd12b3c92ae40e3ad9c6f2546a839bc5628
Sep 12 03:11:12.420702 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:12.420671    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:11:12.420807 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:12.420769    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:11:12.420874 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:12.420819    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:11:13.090084 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:13.090053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" event={"ID":"037a1dc0-54c7-460b-9968-f92128742f37","Type":"ContainerStarted","Data":"afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"}
Sep 12 03:11:13.090251 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:13.090120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" event={"ID":"037a1dc0-54c7-460b-9968-f92128742f37","Type":"ContainerStarted","Data":"63c235d03f9866239e59deeebf60cbd12b3c92ae40e3ad9c6f2546a839bc5628"}
Sep 12 03:11:13.111354 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:13.111306    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podStartSLOduration=2.111292453 podStartE2EDuration="2.111292453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:11:13.10959486 +0000 UTC m=+2091.392508439" watchObservedRunningTime="2026-09-12 03:11:13.111292453 +0000 UTC m=+2091.394206033"
Sep 12 03:11:13.125782 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:13.125758    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:13.130750 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:13.130732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:14.095610 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:14.095570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:14.096841 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:14.096820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:14.434820 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:14.434788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:15.068664 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:15.068632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:15.166806 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:15.166766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:15.272998 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:15.272964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:15.946106 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:15.946073    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:16.683424 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:16.683390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:17.349012 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:17.348975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:18.158908 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:18.158877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:18.239483 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:18.239435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:18.297907 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:18.297874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:18.612577 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:18.612548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:18.713690 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:18.713655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:22.304872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:22.304844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:11:22.308137 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:22.308114    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:11:33.192208 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:33.192174    2579 generic.go:356] "Generic (PLEG): container finished" podID="037a1dc0-54c7-460b-9968-f92128742f37" containerID="afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58" exitCode=0
Sep 12 03:11:33.192645 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:33.192244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" event={"ID":"037a1dc0-54c7-460b-9968-f92128742f37","Type":"ContainerDied","Data":"afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"}
Sep 12 03:11:33.192699 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:33.192660    2579 scope.go:117] "RemoveContainer" containerID="afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"
Sep 12 03:11:33.193803 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:33.193778    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:11:33.193873 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:33.193845    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:11:33.193910 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:33.193875    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 12 03:11:34.199195 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:34.199161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" event={"ID":"037a1dc0-54c7-460b-9968-f92128742f37","Type":"ContainerStarted","Data":"f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953"}
Sep 12 03:11:35.125608 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:35.125575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:35.125833 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:35.125811    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 12 03:11:35.125882 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:35.125861    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 12 03:11:36.125405 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:36.125370    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 12 03:11:36.125834 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:36.125429    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 12 03:11:37.125348 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:37.125310    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 12 03:11:37.125552 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:37.125377    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 12 03:11:38.126085 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:38.126046    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 12 03:11:38.126473 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:38.126105    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 12 03:11:38.929558 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:38.929520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:39.126299 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:39.126246    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 12 03:11:39.126858 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:39.126308    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 12 03:11:39.634043 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:39.634012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:40.129849 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:40.129819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:40.130227 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:40.130099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:40.130746 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:40.130727    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:11:40.843581 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:40.843549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:41.478333 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:41.478302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:42.228699 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:42.228665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:42.925838 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:42.925808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:43.677132 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:43.677100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:44.333311 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:44.333280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:45.272605 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:45.272574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:46.964300 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:46.964249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:47.673546 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:47.673511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:48.359583 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:48.359546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:49.072317 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:49.072287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:49.778339 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:49.778297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:50.756305 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:50.756256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:56.879075 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:56.879040    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:56.979163 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:56.979137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:57.072410 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:57.072376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:57.129699 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:57.129607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:57.452431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:57.452348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:57.557831 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:57.557788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:58.253839 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:58.253814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:59.163118 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:59.163085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:11:59.790402 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:11:59.790376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:00.452219 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:00.452186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:01.846661 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:01.846632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:03.259740 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:03.259682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:03.362301 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:03.362268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:03.564146 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:03.564053    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"]
Sep 12 03:12:03.564347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:03.564319    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy" containerID="cri-o://f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953" gracePeriod=30
Sep 12 03:12:03.587304 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:03.587257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:07.570342 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:07.570310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:08.343785 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.343752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:08.809349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.809328    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:12:08.931837 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.931805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-data\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.931837 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.931844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-istio-token\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.931885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037a1dc0-54c7-460b-9968-f92128742f37-istiod-ca-cert\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.931903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-envoy\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.931924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-credential-socket\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.931998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-certs\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932029    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-kube-api-access-l94dj\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932409 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-data" (OuterVolumeSpecName: "istio-data") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:08.932506 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-socket\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932555 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037a1dc0-54c7-460b-9968-f92128742f37-istio-podinfo\") pod \"037a1dc0-54c7-460b-9968-f92128742f37\" (UID: \"037a1dc0-54c7-460b-9968-f92128742f37\") "
Sep 12 03:12:08.932555 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:08.932832 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:08.932904 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.932834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.933147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037a1dc0-54c7-460b-9968-f92128742f37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.934413    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.934436    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.934471    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.934491    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037a1dc0-54c7-460b-9968-f92128742f37-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.934506    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:08.935337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.935214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:12:08.935775 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.935403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-istio-token" (OuterVolumeSpecName: "istio-token") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:12:08.935775 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.935428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-kube-api-access-l94dj" (OuterVolumeSpecName: "kube-api-access-l94dj") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "kube-api-access-l94dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:12:08.939497 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:08.937015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/037a1dc0-54c7-460b-9968-f92128742f37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "037a1dc0-54c7-460b-9968-f92128742f37" (UID: "037a1dc0-54c7-460b-9968-f92128742f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:12:09.035349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.035312    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94dj\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-kube-api-access-l94dj\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:09.035349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.035345    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037a1dc0-54c7-460b-9968-f92128742f37-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:09.035349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.035355    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037a1dc0-54c7-460b-9968-f92128742f37-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:09.035611 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.035363    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037a1dc0-54c7-460b-9968-f92128742f37-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:09.175322 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.175291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:09.380686 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.380652    2579 generic.go:356] "Generic (PLEG): container finished" podID="037a1dc0-54c7-460b-9968-f92128742f37" containerID="f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953" exitCode=0
Sep 12 03:12:09.380859 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.380722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"
Sep 12 03:12:09.380859 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.380725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" event={"ID":"037a1dc0-54c7-460b-9968-f92128742f37","Type":"ContainerDied","Data":"f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953"}
Sep 12 03:12:09.380859 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.380778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml" event={"ID":"037a1dc0-54c7-460b-9968-f92128742f37","Type":"ContainerDied","Data":"63c235d03f9866239e59deeebf60cbd12b3c92ae40e3ad9c6f2546a839bc5628"}
Sep 12 03:12:09.380859 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.380800    2579 scope.go:117] "RemoveContainer" containerID="f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953"
Sep 12 03:12:09.390593 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.390492    2579 scope.go:117] "RemoveContainer" containerID="afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"
Sep 12 03:12:09.399432 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.399402    2579 scope.go:117] "RemoveContainer" containerID="f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953"
Sep 12 03:12:09.399747 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:12:09.399728    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953\": container with ID starting with f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953 not found: ID does not exist" containerID="f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953"
Sep 12 03:12:09.399814 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.399757    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953"} err="failed to get container status \"f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953\": rpc error: code = NotFound desc = could not find container \"f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953\": container with ID starting with f578381fac962b2972b025170cc75c3afdad77e341c303cb48de13af49f87953 not found: ID does not exist"
Sep 12 03:12:09.399814 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.399775    2579 scope.go:117] "RemoveContainer" containerID="afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"
Sep 12 03:12:09.400019 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:12:09.400005    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58\": container with ID starting with afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58 not found: ID does not exist" containerID="afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"
Sep 12 03:12:09.400069 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.400037    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58"} err="failed to get container status \"afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58\": rpc error: code = NotFound desc = could not find container \"afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58\": container with ID starting with afdf9be1004382732281b43c6e701eb5a92681b8fdeb17ca5559f1dd82049a58 not found: ID does not exist"
Sep 12 03:12:09.405461 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.405421    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"]
Sep 12 03:12:09.409057 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.409038    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78f530-openshift-default-9f546786b-f5xml"]
Sep 12 03:12:09.827850 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:09.827767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:10.246024 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:10.245993    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037a1dc0-54c7-460b-9968-f92128742f37" path="/var/lib/kubelet/pods/037a1dc0-54c7-460b-9968-f92128742f37/volumes"
Sep 12 03:12:10.553623 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:10.553548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:11.239999 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:11.239964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.119221 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.119184    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"]
Sep 12 03:12:23.119830 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.119583    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" podUID="595fdf88-87c7-42dc-b860-4e4d6210d69a" containerName="maas-api" containerID="cri-o://4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e" gracePeriod=30
Sep 12 03:12:23.147352 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.147318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.214927 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.214900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.273127 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.273089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.337529 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.337498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.385023 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.384989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:12:23.389525 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.389499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.453354 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.453324    2579 generic.go:356] "Generic (PLEG): container finished" podID="595fdf88-87c7-42dc-b860-4e4d6210d69a" containerID="4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e" exitCode=0
Sep 12 03:12:23.453551 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.453384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" event={"ID":"595fdf88-87c7-42dc-b860-4e4d6210d69a","Type":"ContainerDied","Data":"4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e"}
Sep 12 03:12:23.453551 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.453426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb" event={"ID":"595fdf88-87c7-42dc-b860-4e4d6210d69a","Type":"ContainerDied","Data":"2992e8ef3a0a7813d8a347c167cc5cac36b13d9d4829d1a4443a770dc90f5f7b"}
Sep 12 03:12:23.453551 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.453468    2579 scope.go:117] "RemoveContainer" containerID="4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e"
Sep 12 03:12:23.453551 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.453384    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"
Sep 12 03:12:23.469996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.469962    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls\") pod \"595fdf88-87c7-42dc-b860-4e4d6210d69a\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") "
Sep 12 03:12:23.469996 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.470010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595fdf88-87c7-42dc-b860-4e4d6210d69a-openshift-service-ca\") pod \"595fdf88-87c7-42dc-b860-4e4d6210d69a\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") "
Sep 12 03:12:23.470280 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.470052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-metrics-certs\") pod \"595fdf88-87c7-42dc-b860-4e4d6210d69a\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") "
Sep 12 03:12:23.470280 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.470166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9rbt\" (UniqueName: \"kubernetes.io/projected/595fdf88-87c7-42dc-b860-4e4d6210d69a-kube-api-access-s9rbt\") pod \"595fdf88-87c7-42dc-b860-4e4d6210d69a\" (UID: \"595fdf88-87c7-42dc-b860-4e4d6210d69a\") "
Sep 12 03:12:23.470839 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.470791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595fdf88-87c7-42dc-b860-4e4d6210d69a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "595fdf88-87c7-42dc-b860-4e4d6210d69a" (UID: "595fdf88-87c7-42dc-b860-4e4d6210d69a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:12:23.471721 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.471694    2579 scope.go:117] "RemoveContainer" containerID="4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e"
Sep 12 03:12:23.472341 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:12:23.472260    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e\": container with ID starting with 4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e not found: ID does not exist" containerID="4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e"
Sep 12 03:12:23.472341 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.472305    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e"} err="failed to get container status \"4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e\": rpc error: code = NotFound desc = could not find container \"4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e\": container with ID starting with 4fe5c872d46b4a5339ac7f46f002d8fe685fdf28d07bd9a7dc6c84d80c9aa95e not found: ID does not exist"
Sep 12 03:12:23.474227 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.474196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595fdf88-87c7-42dc-b860-4e4d6210d69a-kube-api-access-s9rbt" (OuterVolumeSpecName: "kube-api-access-s9rbt") pod "595fdf88-87c7-42dc-b860-4e4d6210d69a" (UID: "595fdf88-87c7-42dc-b860-4e4d6210d69a"). InnerVolumeSpecName "kube-api-access-s9rbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:12:23.474551 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.474514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "595fdf88-87c7-42dc-b860-4e4d6210d69a" (UID: "595fdf88-87c7-42dc-b860-4e4d6210d69a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:12:23.474796 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.474776    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "595fdf88-87c7-42dc-b860-4e4d6210d69a" (UID: "595fdf88-87c7-42dc-b860-4e4d6210d69a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:12:23.571162 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.571116    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-maas-api-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:23.571348 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.571174    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595fdf88-87c7-42dc-b860-4e4d6210d69a-openshift-service-ca\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:23.571348 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.571188    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595fdf88-87c7-42dc-b860-4e4d6210d69a-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:23.571348 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.571202    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rbt\" (UniqueName: \"kubernetes.io/projected/595fdf88-87c7-42dc-b860-4e4d6210d69a-kube-api-access-s9rbt\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:12:23.733423 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.733333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:23.776270 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.776233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"]
Sep 12 03:12:23.782359 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.782325    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad5a1b-56b5974b55-686zb"]
Sep 12 03:12:23.835443 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:23.835401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:24.245794 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:24.245760    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595fdf88-87c7-42dc-b860-4e4d6210d69a" path="/var/lib/kubelet/pods/595fdf88-87c7-42dc-b860-4e4d6210d69a/volumes"
Sep 12 03:12:24.522891 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:24.522805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:25.162892 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:25.162855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:25.952425 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:25.952388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:28.764678 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:28.764645    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:12:44.315110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:12:44.315079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:02.185991 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:02.185954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:02.276143 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:02.276116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:04.995188 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:04.995156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:05.693822 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:05.693785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:11.509227 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:11.509195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:11.571349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:11.571316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:11.793758 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:11.793671    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"]
Sep 12 03:13:11.793987 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:11.793933    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy" containerID="cri-o://f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d" gracePeriod=30
Sep 12 03:13:11.816992 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:11.816967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:12.257616 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:12.257592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:12.918763 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:12.918724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:13.648886 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:13.648851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:14.331778 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:14.331719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:14.975094 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:14.975065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:17.042931 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.042906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:13:17.154339 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-data\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154339 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-certs\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154387    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-credential-socket\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-socket\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154442    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-envoy\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154523    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-token\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154613 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154577    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-podinfo\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154884 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istiod-ca-cert\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.154884 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:17.154884 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc482\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-kube-api-access-xc482\") pod \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\" (UID: \"719c13dd-ffcf-407a-9f61-6c67daa22e3b\") "
Sep 12 03:13:17.155105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.155059    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.155105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:17.155105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.154755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:17.155105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.155084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:17.155105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.155022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:17.156940 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.156917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:13:17.157075 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.156978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:17.157075 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.156986    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-kube-api-access-xc482" (OuterVolumeSpecName: "kube-api-access-xc482") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "kube-api-access-xc482". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:17.157219 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.157193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "719c13dd-ffcf-407a-9f61-6c67daa22e3b" (UID: "719c13dd-ffcf-407a-9f61-6c67daa22e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:17.255624 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255602    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255624 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255621    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255774 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255630    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc482\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-kube-api-access-xc482\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255774 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255639    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255774 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255647    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255774 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255654    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255774 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255662    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.255774 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.255670    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719c13dd-ffcf-407a-9f61-6c67daa22e3b-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:17.720915 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.720879    2579 generic.go:356] "Generic (PLEG): container finished" podID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerID="f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d" exitCode=0
Sep 12 03:13:17.721117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.720934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" event={"ID":"719c13dd-ffcf-407a-9f61-6c67daa22e3b","Type":"ContainerDied","Data":"f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d"}
Sep 12 03:13:17.721117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.720938    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"
Sep 12 03:13:17.721117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.720975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj" event={"ID":"719c13dd-ffcf-407a-9f61-6c67daa22e3b","Type":"ContainerDied","Data":"08e00225d2c33cc0cfedbb3304a5cd7767ad5281fbf8c0fddc0b452efb886216"}
Sep 12 03:13:17.721117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.720995    2579 scope.go:117] "RemoveContainer" containerID="f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d"
Sep 12 03:13:17.730471 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.730435    2579 scope.go:117] "RemoveContainer" containerID="1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"
Sep 12 03:13:17.738644 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.738625    2579 scope.go:117] "RemoveContainer" containerID="f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d"
Sep 12 03:13:17.738900 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:17.738878    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d\": container with ID starting with f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d not found: ID does not exist" containerID="f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d"
Sep 12 03:13:17.738945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.738913    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d"} err="failed to get container status \"f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d\": rpc error: code = NotFound desc = could not find container \"f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d\": container with ID starting with f49559a2354bf9adfb78e9a34c47d0eb358742394862214489bc966f8b7f3a5d not found: ID does not exist"
Sep 12 03:13:17.738988 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.738951    2579 scope.go:117] "RemoveContainer" containerID="1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"
Sep 12 03:13:17.739181 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:17.739158    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4\": container with ID starting with 1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4 not found: ID does not exist" containerID="1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"
Sep 12 03:13:17.739220 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.739191    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4"} err="failed to get container status \"1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4\": rpc error: code = NotFound desc = could not find container \"1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4\": container with ID starting with 1102b03961608a15cd132a1660a370a984ecfc076eba42ae47aa3b21e08767b4 not found: ID does not exist"
Sep 12 03:13:17.741314 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.741294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"]
Sep 12 03:13:17.746654 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:17.746634    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad5a1b-openshift-default-5c467d9f78-8xqfj"]
Sep 12 03:13:18.245628 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:18.245593    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" path="/var/lib/kubelet/pods/719c13dd-ffcf-407a-9f61-6c67daa22e3b/volumes"
Sep 12 03:13:22.769033 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:22.768941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:22.791032 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:22.790999    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"]
Sep 12 03:13:22.791303 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:22.791269    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x" podUID="15b3291a-b81d-4795-b464-d1b729ea2e56" containerName="payload-processing" containerID="cri-o://e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5" gracePeriod=30
Sep 12 03:13:22.870493 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:22.870435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:22.950405 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:22.950369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:23.007721 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.007685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:23.055719 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.055687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:13:23.207635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.207605    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b3291a-b81d-4795-b464-d1b729ea2e56-plugins-config-volume\") pod \"15b3291a-b81d-4795-b464-d1b729ea2e56\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") "
Sep 12 03:13:23.207847 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.207652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbhcp\" (UniqueName: \"kubernetes.io/projected/15b3291a-b81d-4795-b464-d1b729ea2e56-kube-api-access-zbhcp\") pod \"15b3291a-b81d-4795-b464-d1b729ea2e56\" (UID: \"15b3291a-b81d-4795-b464-d1b729ea2e56\") "
Sep 12 03:13:23.208045 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.208023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b3291a-b81d-4795-b464-d1b729ea2e56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15b3291a-b81d-4795-b464-d1b729ea2e56" (UID: "15b3291a-b81d-4795-b464-d1b729ea2e56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:23.209884 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.209863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b3291a-b81d-4795-b464-d1b729ea2e56-kube-api-access-zbhcp" (OuterVolumeSpecName: "kube-api-access-zbhcp") pod "15b3291a-b81d-4795-b464-d1b729ea2e56" (UID: "15b3291a-b81d-4795-b464-d1b729ea2e56"). InnerVolumeSpecName "kube-api-access-zbhcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:23.309146 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.309069    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b3291a-b81d-4795-b464-d1b729ea2e56-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:23.309146 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.309092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhcp\" (UniqueName: \"kubernetes.io/projected/15b3291a-b81d-4795-b464-d1b729ea2e56-kube-api-access-zbhcp\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:23.357336 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.357309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:23.410562 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.410531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:23.755491 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.755427    2579 generic.go:356] "Generic (PLEG): container finished" podID="15b3291a-b81d-4795-b464-d1b729ea2e56" containerID="e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5" exitCode=0
Sep 12 03:13:23.755707 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.755510    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"
Sep 12 03:13:23.755707 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.755514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x" event={"ID":"15b3291a-b81d-4795-b464-d1b729ea2e56","Type":"ContainerDied","Data":"e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5"}
Sep 12 03:13:23.755707 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.755617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x" event={"ID":"15b3291a-b81d-4795-b464-d1b729ea2e56","Type":"ContainerDied","Data":"8296564e62ce21f2784c5ffb460c2e91107a2381bb2ea8bb84673ecedcec7be5"}
Sep 12 03:13:23.755707 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.755656    2579 scope.go:117] "RemoveContainer" containerID="e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5"
Sep 12 03:13:23.765806 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.765684    2579 scope.go:117] "RemoveContainer" containerID="e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5"
Sep 12 03:13:23.766111 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:23.766020    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5\": container with ID starting with e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5 not found: ID does not exist" containerID="e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5"
Sep 12 03:13:23.766111 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.766059    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5"} err="failed to get container status \"e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5\": rpc error: code = NotFound desc = could not find container \"e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5\": container with ID starting with e035047515c9d3d777a336846fc6e6400c89151588d4f35ae960d680ed7852f5 not found: ID does not exist"
Sep 12 03:13:23.780085 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.780060    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"]
Sep 12 03:13:23.783034 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:23.783015    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-27bf83-75bcd84559-tkb7x"]
Sep 12 03:13:24.039413 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:24.039333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:24.246438 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:24.246410    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15b3291a-b81d-4795-b464-d1b729ea2e56" path="/var/lib/kubelet/pods/15b3291a-b81d-4795-b464-d1b729ea2e56/volumes"
Sep 12 03:13:24.929467 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:24.929423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:29.134592 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:29.134560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:29.233472 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:29.233423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:29.391314 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:29.391228    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"]
Sep 12 03:13:29.391552 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:29.391498    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy" containerID="cri-o://4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a" gracePeriod=30
Sep 12 03:13:29.417133 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:29.417107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:30.443514 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:30.443481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:31.075334 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:31.075303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:31.859554 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:31.859523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:34.643961 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.643933    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:13:34.712718 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8afb8f-8681-4466-99cb-0229d84deacd-istiod-ca-cert\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712718 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-certs\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-podinfo\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-data\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7nkt\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-kube-api-access-s7nkt\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-token\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-socket\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712871    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-envoy\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.712948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.712889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-credential-socket\") pod \"fc8afb8f-8681-4466-99cb-0229d84deacd\" (UID: \"fc8afb8f-8681-4466-99cb-0229d84deacd\") "
Sep 12 03:13:34.713295 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.713110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:34.713295 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.713123    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:34.713295 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.713128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8afb8f-8681-4466-99cb-0229d84deacd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:34.713437 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.713294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:34.713617 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.713507    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:34.715220 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.715193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 03:13:34.715318 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.715193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:13:34.715318 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.715226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-kube-api-access-s7nkt" (OuterVolumeSpecName: "kube-api-access-s7nkt") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "kube-api-access-s7nkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:34.715318 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.715300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc8afb8f-8681-4466-99cb-0229d84deacd" (UID: "fc8afb8f-8681-4466-99cb-0229d84deacd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:34.813880 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813847    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nkt\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-kube-api-access-s7nkt\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.813880 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813873    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.813880 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813883    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.813880 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813891    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-envoy\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.814238 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813899    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-credential-socket\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.814238 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813907    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8afb8f-8681-4466-99cb-0229d84deacd-istiod-ca-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.814238 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813914    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-workload-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.814238 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813921    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-podinfo\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.814238 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.813928    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8afb8f-8681-4466-99cb-0229d84deacd-istio-data\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:34.818503 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.818472    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerID="4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a" exitCode=0
Sep 12 03:13:34.818713 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.818541    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"
Sep 12 03:13:34.818713 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.818542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" event={"ID":"fc8afb8f-8681-4466-99cb-0229d84deacd","Type":"ContainerDied","Data":"4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a"}
Sep 12 03:13:34.818713 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.818618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp" event={"ID":"fc8afb8f-8681-4466-99cb-0229d84deacd","Type":"ContainerDied","Data":"0d112e9cc8050974aa86e2c1c42fa75962f65dfe93afa6f261f5faade4a3d4c6"}
Sep 12 03:13:34.818713 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.818641    2579 scope.go:117] "RemoveContainer" containerID="4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a"
Sep 12 03:13:34.827682 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.827662    2579 scope.go:117] "RemoveContainer" containerID="a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"
Sep 12 03:13:34.835431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.835411    2579 scope.go:117] "RemoveContainer" containerID="4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a"
Sep 12 03:13:34.835719 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:34.835699    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a\": container with ID starting with 4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a not found: ID does not exist" containerID="4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a"
Sep 12 03:13:34.835764 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.835728    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a"} err="failed to get container status \"4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a\": rpc error: code = NotFound desc = could not find container \"4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a\": container with ID starting with 4ec8ef4b72140699ab76d08a47f6a7c2a35f77e64ecea464d974bf9f98648d8a not found: ID does not exist"
Sep 12 03:13:34.835764 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.835746    2579 scope.go:117] "RemoveContainer" containerID="a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"
Sep 12 03:13:34.836004 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:34.835982    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348\": container with ID starting with a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348 not found: ID does not exist" containerID="a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"
Sep 12 03:13:34.836089 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.836008    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348"} err="failed to get container status \"a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348\": rpc error: code = NotFound desc = could not find container \"a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348\": container with ID starting with a7a62c70888ee1e82b571f7b0916776168d2e1444a7c3fe1e20fc6fa7f666348 not found: ID does not exist"
Sep 12 03:13:34.839581 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.839561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"]
Sep 12 03:13:34.842921 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:34.842901    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-27bf83-openshift-default-575b4cd9f6-65hfp"]
Sep 12 03:13:36.245586 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:36.245555    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" path="/var/lib/kubelet/pods/fc8afb8f-8681-4466-99cb-0229d84deacd/volumes"
Sep 12 03:13:40.253795 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.253761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:40.278584 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.278553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"]
Sep 12 03:13:40.278844 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.278795    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj" podUID="447f380b-3ba0-4a91-a5a3-d987dad090bc" containerName="payload-processing" containerID="cri-o://7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40" gracePeriod=30
Sep 12 03:13:40.356217 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.356178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:40.437241 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.437209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:40.490825 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.490792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:40.546184 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.546153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:13:40.553115 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.553091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnsmd\" (UniqueName: \"kubernetes.io/projected/447f380b-3ba0-4a91-a5a3-d987dad090bc-kube-api-access-cnsmd\") pod \"447f380b-3ba0-4a91-a5a3-d987dad090bc\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") "
Sep 12 03:13:40.553272 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.553210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/447f380b-3ba0-4a91-a5a3-d987dad090bc-plugins-config-volume\") pod \"447f380b-3ba0-4a91-a5a3-d987dad090bc\" (UID: \"447f380b-3ba0-4a91-a5a3-d987dad090bc\") "
Sep 12 03:13:40.553564 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.553544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447f380b-3ba0-4a91-a5a3-d987dad090bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "447f380b-3ba0-4a91-a5a3-d987dad090bc" (UID: "447f380b-3ba0-4a91-a5a3-d987dad090bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:40.555270 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.555249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447f380b-3ba0-4a91-a5a3-d987dad090bc-kube-api-access-cnsmd" (OuterVolumeSpecName: "kube-api-access-cnsmd") pod "447f380b-3ba0-4a91-a5a3-d987dad090bc" (UID: "447f380b-3ba0-4a91-a5a3-d987dad090bc"). InnerVolumeSpecName "kube-api-access-cnsmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:40.654534 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.654492    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/447f380b-3ba0-4a91-a5a3-d987dad090bc-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:40.654534 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.654524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnsmd\" (UniqueName: \"kubernetes.io/projected/447f380b-3ba0-4a91-a5a3-d987dad090bc-kube-api-access-cnsmd\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:40.832806 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.832722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:40.849724 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.849695    2579 generic.go:356] "Generic (PLEG): container finished" podID="447f380b-3ba0-4a91-a5a3-d987dad090bc" containerID="7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40" exitCode=0
Sep 12 03:13:40.849896 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.849731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj" event={"ID":"447f380b-3ba0-4a91-a5a3-d987dad090bc","Type":"ContainerDied","Data":"7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40"}
Sep 12 03:13:40.849896 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.849742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"
Sep 12 03:13:40.849896 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.849809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj" event={"ID":"447f380b-3ba0-4a91-a5a3-d987dad090bc","Type":"ContainerDied","Data":"e286d8d7326843693b856720aaed36151b232ed49c214391abb213e5e5278f4a"}
Sep 12 03:13:40.849896 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.849834    2579 scope.go:117] "RemoveContainer" containerID="7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40"
Sep 12 03:13:40.859026 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.859005    2579 scope.go:117] "RemoveContainer" containerID="7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40"
Sep 12 03:13:40.859306 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:40.859290    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40\": container with ID starting with 7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40 not found: ID does not exist" containerID="7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40"
Sep 12 03:13:40.859354 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.859314    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40"} err="failed to get container status \"7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40\": rpc error: code = NotFound desc = could not find container \"7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40\": container with ID starting with 7d90f9b720dc33eef957ef66d9b8782478df848a7789fe1b13c917e957b5bd40 not found: ID does not exist"
Sep 12 03:13:40.887561 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.887524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"]
Sep 12 03:13:40.898463 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.898417    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f91169-69b8988787-k69lj"]
Sep 12 03:13:40.971415 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:40.971382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:41.652296 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:41.652259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:42.245598 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:42.245568    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="447f380b-3ba0-4a91-a5a3-d987dad090bc" path="/var/lib/kubelet/pods/447f380b-3ba0-4a91-a5a3-d987dad090bc/volumes"
Sep 12 03:13:46.797622 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:46.797537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:46.903299 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:46.903267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:47.109439 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:47.109409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:47.570683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:47.570591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:48.334658 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:48.334625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:49.127528 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:49.127493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:57.919325 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:57.919282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:57.947442 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:57.947407    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"]
Sep 12 03:13:57.947665 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:57.947622    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" podUID="e82c544c-60fc-471d-8b66-51a94400b15f" containerName="payload-processing" containerID="cri-o://b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc" gracePeriod=30
Sep 12 03:13:57.985811 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:57.985785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:57.995991 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:57.995961    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" podUID="e82c544c-60fc-471d-8b66-51a94400b15f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.61:9004: connect: connection refused"
Sep 12 03:13:58.086347 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.086311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:58.137896 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.137872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:58.207330 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.207305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:13:58.307685 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.307658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/e82c544c-60fc-471d-8b66-51a94400b15f-kube-api-access-qgpv6\") pod \"e82c544c-60fc-471d-8b66-51a94400b15f\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") "
Sep 12 03:13:58.307842 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.307733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82c544c-60fc-471d-8b66-51a94400b15f-plugins-config-volume\") pod \"e82c544c-60fc-471d-8b66-51a94400b15f\" (UID: \"e82c544c-60fc-471d-8b66-51a94400b15f\") "
Sep 12 03:13:58.308054 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.308032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82c544c-60fc-471d-8b66-51a94400b15f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e82c544c-60fc-471d-8b66-51a94400b15f" (UID: "e82c544c-60fc-471d-8b66-51a94400b15f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:13:58.309892 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.309872    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82c544c-60fc-471d-8b66-51a94400b15f-kube-api-access-qgpv6" (OuterVolumeSpecName: "kube-api-access-qgpv6") pod "e82c544c-60fc-471d-8b66-51a94400b15f" (UID: "e82c544c-60fc-471d-8b66-51a94400b15f"). InnerVolumeSpecName "kube-api-access-qgpv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:13:58.408439 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.408405    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/e82c544c-60fc-471d-8b66-51a94400b15f-kube-api-access-qgpv6\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:58.408439 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.408433    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82c544c-60fc-471d-8b66-51a94400b15f-plugins-config-volume\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:13:58.525835 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.525758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:58.626926 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.626893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:13:58.937828 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.937798    2579 generic.go:356] "Generic (PLEG): container finished" podID="e82c544c-60fc-471d-8b66-51a94400b15f" containerID="b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc" exitCode=0
Sep 12 03:13:58.938218 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.937839    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"
Sep 12 03:13:58.938218 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.937860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" event={"ID":"e82c544c-60fc-471d-8b66-51a94400b15f","Type":"ContainerDied","Data":"b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc"}
Sep 12 03:13:58.938218 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.937907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg" event={"ID":"e82c544c-60fc-471d-8b66-51a94400b15f","Type":"ContainerDied","Data":"c4ce785618022f24e7f2355abc6da3a9fa3d4dd60a3ac0c9f0b7efe23d973977"}
Sep 12 03:13:58.938218 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.937930    2579 scope.go:117] "RemoveContainer" containerID="b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc"
Sep 12 03:13:58.946914 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.946897    2579 scope.go:117] "RemoveContainer" containerID="b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc"
Sep 12 03:13:58.947157 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:13:58.947142    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc\": container with ID starting with b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc not found: ID does not exist" containerID="b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc"
Sep 12 03:13:58.947196 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.947165    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc"} err="failed to get container status \"b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc\": rpc error: code = NotFound desc = could not find container \"b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc\": container with ID starting with b2d385c0820943174d129432c9ed983e83276a9442bdce3bfdbd05b1811ef1dc not found: ID does not exist"
Sep 12 03:13:58.972403 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.972373    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"]
Sep 12 03:13:58.980520 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:13:58.980496    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e401bd-95ff988b5-kctgg"]
Sep 12 03:14:00.245238 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:00.245202    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82c544c-60fc-471d-8b66-51a94400b15f" path="/var/lib/kubelet/pods/e82c544c-60fc-471d-8b66-51a94400b15f/volumes"
Sep 12 03:14:04.222930 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:04.222896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:14:04.288581 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:04.288545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:14:04.588079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:04.587999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:14:07.089332 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.089291    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5848789547-brjgr"]
Sep 12 03:14:07.089853 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.089545    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5848789547-brjgr" podUID="2003500d-8b2d-433c-83f6-ffd91f67c6f9" containerName="manager" containerID="cri-o://cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857" gracePeriod=30
Sep 12 03:14:07.341874 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.341819    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 03:14:07.489337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.489308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-webhook-cert\") pod \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") "
Sep 12 03:14:07.489548 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.489363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvssk\" (UniqueName: \"kubernetes.io/projected/2003500d-8b2d-433c-83f6-ffd91f67c6f9-kube-api-access-gvssk\") pod \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") "
Sep 12 03:14:07.489548 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.489441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-metrics-certs\") pod \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\" (UID: \"2003500d-8b2d-433c-83f6-ffd91f67c6f9\") "
Sep 12 03:14:07.491568 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.491538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2003500d-8b2d-433c-83f6-ffd91f67c6f9-kube-api-access-gvssk" (OuterVolumeSpecName: "kube-api-access-gvssk") pod "2003500d-8b2d-433c-83f6-ffd91f67c6f9" (UID: "2003500d-8b2d-433c-83f6-ffd91f67c6f9"). InnerVolumeSpecName "kube-api-access-gvssk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:14:07.491679 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.491598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2003500d-8b2d-433c-83f6-ffd91f67c6f9" (UID: "2003500d-8b2d-433c-83f6-ffd91f67c6f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:14:07.491679 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.491606    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2003500d-8b2d-433c-83f6-ffd91f67c6f9" (UID: "2003500d-8b2d-433c-83f6-ffd91f67c6f9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:14:07.590431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.590396    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:14:07.590431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.590426    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2003500d-8b2d-433c-83f6-ffd91f67c6f9-webhook-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:14:07.590431 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.590435    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvssk\" (UniqueName: \"kubernetes.io/projected/2003500d-8b2d-433c-83f6-ffd91f67c6f9-kube-api-access-gvssk\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:14:07.988718 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.988685    2579 generic.go:356] "Generic (PLEG): container finished" podID="2003500d-8b2d-433c-83f6-ffd91f67c6f9" containerID="cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857" exitCode=0
Sep 12 03:14:07.988945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.988742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-brjgr"
Sep 12 03:14:07.988945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.988762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-brjgr" event={"ID":"2003500d-8b2d-433c-83f6-ffd91f67c6f9","Type":"ContainerDied","Data":"cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857"}
Sep 12 03:14:07.988945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.988833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-brjgr" event={"ID":"2003500d-8b2d-433c-83f6-ffd91f67c6f9","Type":"ContainerDied","Data":"7a0a2f093f749851c65d8a2b0fbba62c1c797f62137df5b21ba8f5a7a7b6ea66"}
Sep 12 03:14:07.988945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.988855    2579 scope.go:117] "RemoveContainer" containerID="cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857"
Sep 12 03:14:07.998565 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.998548    2579 scope.go:117] "RemoveContainer" containerID="cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857"
Sep 12 03:14:07.998894 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:14:07.998873    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857\": container with ID starting with cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857 not found: ID does not exist" containerID="cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857"
Sep 12 03:14:07.998954 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:07.998904    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857"} err="failed to get container status \"cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857\": rpc error: code = NotFound desc = could not find container \"cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857\": container with ID starting with cc3dee2e69a88473e11b31e1fecafe76913bac2fb2712496f949a4dd8215d857 not found: ID does not exist"
Sep 12 03:14:08.009873 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:08.009852    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5848789547-brjgr"]
Sep 12 03:14:08.012945 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:08.012924    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5848789547-brjgr"]
Sep 12 03:14:08.246008 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:08.245930    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2003500d-8b2d-433c-83f6-ffd91f67c6f9" path="/var/lib/kubelet/pods/2003500d-8b2d-433c-83f6-ffd91f67c6f9/volumes"
Sep 12 03:14:47.442672 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:14:47.442642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:15:02.865604 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:02.865573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:15:08.810612 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.810567    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5848789547-692lt"]
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811040    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811061    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811097    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2003500d-8b2d-433c-83f6-ffd91f67c6f9" containerName="manager"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2003500d-8b2d-433c-83f6-ffd91f67c6f9" containerName="manager"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811117    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811146    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811155    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811165    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595fdf88-87c7-42dc-b860-4e4d6210d69a" containerName="maas-api"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811174    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="595fdf88-87c7-42dc-b860-4e4d6210d69a" containerName="maas-api"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811182    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447f380b-3ba0-4a91-a5a3-d987dad090bc" containerName="payload-processing"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811190    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="447f380b-3ba0-4a91-a5a3-d987dad090bc" containerName="payload-processing"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811198    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82c544c-60fc-471d-8b66-51a94400b15f" containerName="payload-processing"
Sep 12 03:15:08.811199 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811207    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82c544c-60fc-471d-8b66-51a94400b15f" containerName="payload-processing"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811225    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b3291a-b81d-4795-b464-d1b729ea2e56" containerName="payload-processing"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811232    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b3291a-b81d-4795-b464-d1b729ea2e56" containerName="payload-processing"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811318    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811329    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811340    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82c544c-60fc-471d-8b66-51a94400b15f" containerName="payload-processing"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811350    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="595fdf88-87c7-42dc-b860-4e4d6210d69a" containerName="maas-api"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811357    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811366    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811374    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="037a1dc0-54c7-460b-9968-f92128742f37" containerName="istio-proxy"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811388    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15b3291a-b81d-4795-b464-d1b729ea2e56" containerName="payload-processing"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811398    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="447f380b-3ba0-4a91-a5a3-d987dad090bc" containerName="payload-processing"
Sep 12 03:15:08.812020 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.811406    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2003500d-8b2d-433c-83f6-ffd91f67c6f9" containerName="manager"
Sep 12 03:15:08.817180 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.817148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:08.822136 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.822108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fs74v\""
Sep 12 03:15:08.822266 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.822137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 03:15:08.822266 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.822116    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 03:15:08.822405 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.822265    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 03:15:08.825016 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.824893    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5848789547-692lt"]
Sep 12 03:15:08.919249 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.919217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-metrics-certs\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:08.919432 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.919263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7fgz\" (UniqueName: \"kubernetes.io/projected/8391795c-eb16-48e4-9064-54d2364610cb-kube-api-access-b7fgz\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:08.919432 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:08.919306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-webhook-cert\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.020148 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.020111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-metrics-certs\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.020148 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.020155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7fgz\" (UniqueName: \"kubernetes.io/projected/8391795c-eb16-48e4-9064-54d2364610cb-kube-api-access-b7fgz\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.020428 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.020183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-webhook-cert\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.022743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.022721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-webhook-cert\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.022872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.022759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-metrics-certs\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.028402 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.028382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7fgz\" (UniqueName: \"kubernetes.io/projected/8391795c-eb16-48e4-9064-54d2364610cb-kube-api-access-b7fgz\") pod \"maas-controller-5848789547-692lt\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") " pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.129858 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.129831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:09.400488 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.400443    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5848789547-692lt"]
Sep 12 03:15:09.403242 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:15:09.403215    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8391795c_eb16_48e4_9064_54d2364610cb.slice/crio-3a6ec383996438ffa1216cc79ee9970d58107b31a433ea2403f997e35c1841aa WatchSource:0}: Error finding container 3a6ec383996438ffa1216cc79ee9970d58107b31a433ea2403f997e35c1841aa: Status 404 returned error can't find the container with id 3a6ec383996438ffa1216cc79ee9970d58107b31a433ea2403f997e35c1841aa
Sep 12 03:15:09.405008 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:09.404991    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 03:15:10.275148 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:10.275112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-692lt" event={"ID":"8391795c-eb16-48e4-9064-54d2364610cb","Type":"ContainerStarted","Data":"9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d"}
Sep 12 03:15:10.275582 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:10.275167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-692lt" event={"ID":"8391795c-eb16-48e4-9064-54d2364610cb","Type":"ContainerStarted","Data":"3a6ec383996438ffa1216cc79ee9970d58107b31a433ea2403f997e35c1841aa"}
Sep 12 03:15:10.275582 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:10.275200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:10.292984 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:10.292940    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5848789547-692lt" podStartSLOduration=1.8669239 podStartE2EDuration="2.292925213s" totalImagesPullingTime="426.001313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:15:10.291130926 +0000 UTC m=+2328.574044507" watchObservedRunningTime="2026-09-12 03:15:10.292925213 +0000 UTC m=+2328.575838794"
Sep 12 03:15:21.281094 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:21.281021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:30.562275 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:30.562245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:15:30.666681 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:30.666645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5848789547-692lt"]
Sep 12 03:15:30.666935 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:30.666912    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5848789547-692lt" podUID="8391795c-eb16-48e4-9064-54d2364610cb" containerName="manager" containerID="cri-o://9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d" gracePeriod=30
Sep 12 03:15:30.907729 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:30.907705    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:31.010915 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.010883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7fgz\" (UniqueName: \"kubernetes.io/projected/8391795c-eb16-48e4-9064-54d2364610cb-kube-api-access-b7fgz\") pod \"8391795c-eb16-48e4-9064-54d2364610cb\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") "
Sep 12 03:15:31.011093 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.010980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-metrics-certs\") pod \"8391795c-eb16-48e4-9064-54d2364610cb\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") "
Sep 12 03:15:31.011093 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.011075    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-webhook-cert\") pod \"8391795c-eb16-48e4-9064-54d2364610cb\" (UID: \"8391795c-eb16-48e4-9064-54d2364610cb\") "
Sep 12 03:15:31.013010 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.012980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8391795c-eb16-48e4-9064-54d2364610cb-kube-api-access-b7fgz" (OuterVolumeSpecName: "kube-api-access-b7fgz") pod "8391795c-eb16-48e4-9064-54d2364610cb" (UID: "8391795c-eb16-48e4-9064-54d2364610cb"). InnerVolumeSpecName "kube-api-access-b7fgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:15:31.013010 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.013000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8391795c-eb16-48e4-9064-54d2364610cb" (UID: "8391795c-eb16-48e4-9064-54d2364610cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:15:31.013265 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.013247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8391795c-eb16-48e4-9064-54d2364610cb" (UID: "8391795c-eb16-48e4-9064-54d2364610cb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:15:31.112234 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.112171    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-webhook-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:15:31.112234 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.112224    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7fgz\" (UniqueName: \"kubernetes.io/projected/8391795c-eb16-48e4-9064-54d2364610cb-kube-api-access-b7fgz\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:15:31.112234 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.112239    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8391795c-eb16-48e4-9064-54d2364610cb-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:15:31.373110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.373004    2579 generic.go:356] "Generic (PLEG): container finished" podID="8391795c-eb16-48e4-9064-54d2364610cb" containerID="9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d" exitCode=0
Sep 12 03:15:31.373110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.373080    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-692lt"
Sep 12 03:15:31.373320 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.373075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-692lt" event={"ID":"8391795c-eb16-48e4-9064-54d2364610cb","Type":"ContainerDied","Data":"9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d"}
Sep 12 03:15:31.373320 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.373211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-692lt" event={"ID":"8391795c-eb16-48e4-9064-54d2364610cb","Type":"ContainerDied","Data":"3a6ec383996438ffa1216cc79ee9970d58107b31a433ea2403f997e35c1841aa"}
Sep 12 03:15:31.373320 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.373249    2579 scope.go:117] "RemoveContainer" containerID="9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d"
Sep 12 03:15:31.383243 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.383159    2579 scope.go:117] "RemoveContainer" containerID="9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d"
Sep 12 03:15:31.383598 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:15:31.383570    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d\": container with ID starting with 9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d not found: ID does not exist" containerID="9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d"
Sep 12 03:15:31.383704 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.383609    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d"} err="failed to get container status \"9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d\": rpc error: code = NotFound desc = could not find container \"9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d\": container with ID starting with 9ef41dd1d05cacc3be7adc6af9da1815b6a11f565885cf024b1747d6c8b0c29d not found: ID does not exist"
Sep 12 03:15:31.400077 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.400047    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5848789547-692lt"]
Sep 12 03:15:31.401876 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:31.401856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5848789547-692lt"]
Sep 12 03:15:32.093181 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.093146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:15:32.218778 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.218745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5848789547-j6x7q"]
Sep 12 03:15:32.219079 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219069    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8391795c-eb16-48e4-9064-54d2364610cb" containerName="manager"
Sep 12 03:15:32.219134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219081    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8391795c-eb16-48e4-9064-54d2364610cb" containerName="manager"
Sep 12 03:15:32.219134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219102    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy"
Sep 12 03:15:32.219134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219107    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy"
Sep 12 03:15:32.219134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219115    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy"
Sep 12 03:15:32.219134 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219121    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8afb8f-8681-4466-99cb-0229d84deacd" containerName="istio-proxy"
Sep 12 03:15:32.219333 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8391795c-eb16-48e4-9064-54d2364610cb" containerName="manager"
Sep 12 03:15:32.219333 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.219177    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="719c13dd-ffcf-407a-9f61-6c67daa22e3b" containerName="istio-proxy"
Sep 12 03:15:32.223292 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.223270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.226872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.226830    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 03:15:32.226872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.226853    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fs74v\""
Sep 12 03:15:32.226872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.226857    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 03:15:32.227109 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.226882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 03:15:32.231305 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.231286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5848789547-j6x7q"]
Sep 12 03:15:32.247279 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.247252    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8391795c-eb16-48e4-9064-54d2364610cb" path="/var/lib/kubelet/pods/8391795c-eb16-48e4-9064-54d2364610cb/volumes"
Sep 12 03:15:32.322515 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.322478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hhz\" (UniqueName: \"kubernetes.io/projected/77c96c02-8728-4a76-a6da-b854d74c2666-kube-api-access-z5hhz\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.322683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.322613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77c96c02-8728-4a76-a6da-b854d74c2666-webhook-cert\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.322683 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.322663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c96c02-8728-4a76-a6da-b854d74c2666-metrics-certs\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.423490 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.423432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77c96c02-8728-4a76-a6da-b854d74c2666-webhook-cert\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.423698 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.423500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c96c02-8728-4a76-a6da-b854d74c2666-metrics-certs\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.423698 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.423526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hhz\" (UniqueName: \"kubernetes.io/projected/77c96c02-8728-4a76-a6da-b854d74c2666-kube-api-access-z5hhz\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.425962 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.425939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77c96c02-8728-4a76-a6da-b854d74c2666-webhook-cert\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.426123 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.426103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77c96c02-8728-4a76-a6da-b854d74c2666-metrics-certs\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.432099 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.432077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hhz\" (UniqueName: \"kubernetes.io/projected/77c96c02-8728-4a76-a6da-b854d74c2666-kube-api-access-z5hhz\") pod \"maas-controller-5848789547-j6x7q\" (UID: \"77c96c02-8728-4a76-a6da-b854d74c2666\") " pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.534624 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.534592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:32.805236 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:32.805214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5848789547-j6x7q"]
Sep 12 03:15:32.806795 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:15:32.806768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c96c02_8728_4a76_a6da_b854d74c2666.slice/crio-899f2f055a07eba06b4aa8b7cf01b2268187114f61aac1b3713cdc11c2369e63 WatchSource:0}: Error finding container 899f2f055a07eba06b4aa8b7cf01b2268187114f61aac1b3713cdc11c2369e63: Status 404 returned error can't find the container with id 899f2f055a07eba06b4aa8b7cf01b2268187114f61aac1b3713cdc11c2369e63
Sep 12 03:15:33.385167 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:33.385081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-j6x7q" event={"ID":"77c96c02-8728-4a76-a6da-b854d74c2666","Type":"ContainerStarted","Data":"8e66bfab6b5064f029388712ce70611c5469d91b2fcf990991cc019e9faa3170"}
Sep 12 03:15:33.385167 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:33.385127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5848789547-j6x7q" event={"ID":"77c96c02-8728-4a76-a6da-b854d74c2666","Type":"ContainerStarted","Data":"899f2f055a07eba06b4aa8b7cf01b2268187114f61aac1b3713cdc11c2369e63"}
Sep 12 03:15:33.385623 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:33.385201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:15:33.402510 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:33.402439    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5848789547-j6x7q" podStartSLOduration=1.087256036 podStartE2EDuration="1.402423946s" totalImagesPullingTime="315.16791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:15:33.401251455 +0000 UTC m=+2351.684165037" watchObservedRunningTime="2026-09-12 03:15:33.402423946 +0000 UTC m=+2351.685337526"
Sep 12 03:15:38.730391 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:38.730362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:15:38.830850 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:38.830825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:15:44.392169 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:15:44.392141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5848789547-j6x7q"
Sep 12 03:16:22.331206 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:16:22.331178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:16:22.337259 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:16:22.337240    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvlnt_4d274728-2cb0-4d20-aee9-7841c219f3be/ovn-acl-logging/0.log"
Sep 12 03:16:44.154872 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:16:44.154795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:16:50.994879 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:16:50.994834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:16:51.064988 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:16:51.064952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:16:54.963087 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:16:54.963042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:18:34.919924 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:34.919889    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"]
Sep 12 03:18:34.923019 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:34.922995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:34.925750 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:34.925731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b07066f0-kserve-self-signed-certs\""
Sep 12 03:18:34.932953 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:34.932928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"]
Sep 12 03:18:34.980886 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:34.980848    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:18:35.071726 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.071674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-home\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.071726 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.071723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tls-certs\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.071942 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.071743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-model-cache\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.071942 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.071763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cw4\" (UniqueName: \"kubernetes.io/projected/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-kube-api-access-q5cw4\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.071942 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.071904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tmp-dir\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.072060 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.071943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-dshm\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173035 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.172949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-home\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173035 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.172989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tls-certs\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173035 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-model-cache\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173309 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cw4\" (UniqueName: \"kubernetes.io/projected/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-kube-api-access-q5cw4\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173309 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tmp-dir\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173309 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-dshm\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173502 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-home\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173502 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-model-cache\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.173502 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.173426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tmp-dir\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.175440 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.175420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-dshm\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.175722 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.175704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tls-certs\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.180696 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.180679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cw4\" (UniqueName: \"kubernetes.io/projected/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-kube-api-access-q5cw4\") pod \"e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") " pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.233047 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.233016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:35.503959 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:35.503772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"]
Sep 12 03:18:35.506265 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:18:35.506239    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a8d31e1_c3e1_4f41_9f18_1b455e23e2e2.slice/crio-732f0e697799643b2ee6589e208c880b2e3869bd115ec0215552e88d56c7ffa2 WatchSource:0}: Error finding container 732f0e697799643b2ee6589e208c880b2e3869bd115ec0215552e88d56c7ffa2: Status 404 returned error can't find the container with id 732f0e697799643b2ee6589e208c880b2e3869bd115ec0215552e88d56c7ffa2
Sep 12 03:18:36.208610 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:36.208577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf" event={"ID":"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2","Type":"ContainerStarted","Data":"c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e"}
Sep 12 03:18:36.208610 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:36.208615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf" event={"ID":"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2","Type":"ContainerStarted","Data":"732f0e697799643b2ee6589e208c880b2e3869bd115ec0215552e88d56c7ffa2"}
Sep 12 03:18:36.225034 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:36.224992    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf" podStartSLOduration=2.224978327 podStartE2EDuration="2.224978327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:18:36.224291604 +0000 UTC m=+2534.507205199" watchObservedRunningTime="2026-09-12 03:18:36.224978327 +0000 UTC m=+2534.507891908"
Sep 12 03:18:45.233539 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:45.233505    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:45.233539 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:45.233550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:45.246174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:45.246147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:45.261480 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:45.261425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:18:57.272689 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:18:57.272653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:19:31.728117 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:31.728080    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"]
Sep 12 03:19:31.747484 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:31.747420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:19:32.762213 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:32.762184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:19:46.748235 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:46.748135    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf" podUID="2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" containerName="main" containerID="cri-o://c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e" gracePeriod=45
Sep 12 03:19:46.989219 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:46.989191    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:19:47.099607 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-dshm\") pod \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") "
Sep 12 03:19:47.099607 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099586    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-home\") pod \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") "
Sep 12 03:19:47.099779 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-model-cache\") pod \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") "
Sep 12 03:19:47.099779 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5cw4\" (UniqueName: \"kubernetes.io/projected/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-kube-api-access-q5cw4\") pod \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") "
Sep 12 03:19:47.099779 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tls-certs\") pod \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") "
Sep 12 03:19:47.099779 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tmp-dir\") pod \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\" (UID: \"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2\") "
Sep 12 03:19:47.099990 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099828    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-home" (OuterVolumeSpecName: "home") pod "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" (UID: "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:47.099990 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.099860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" (UID: "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:47.100147 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.100023    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" (UID: "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:47.100147 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.100058    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-home\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:47.100147 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.100069    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-model-cache\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:47.101938 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.101913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-dshm" (OuterVolumeSpecName: "dshm") pod "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" (UID: "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 03:19:47.102047 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.101966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" (UID: "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:19:47.102047 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.101987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-kube-api-access-q5cw4" (OuterVolumeSpecName: "kube-api-access-q5cw4") pod "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" (UID: "2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2"). InnerVolumeSpecName "kube-api-access-q5cw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:19:47.200552 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.200532    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tls-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:47.200552 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.200552    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-tmp-dir\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:47.200703 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.200559    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-dshm\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:47.200703 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.200568    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5cw4\" (UniqueName: \"kubernetes.io/projected/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2-kube-api-access-q5cw4\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:47.539540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.539506    2579 generic.go:356] "Generic (PLEG): container finished" podID="2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" containerID="c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e" exitCode=0
Sep 12 03:19:47.539744 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.539545    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"
Sep 12 03:19:47.539744 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.539595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf" event={"ID":"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2","Type":"ContainerDied","Data":"c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e"}
Sep 12 03:19:47.539744 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.539640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf" event={"ID":"2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2","Type":"ContainerDied","Data":"732f0e697799643b2ee6589e208c880b2e3869bd115ec0215552e88d56c7ffa2"}
Sep 12 03:19:47.539744 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.539662    2579 scope.go:117] "RemoveContainer" containerID="c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e"
Sep 12 03:19:47.548839 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.548824    2579 scope.go:117] "RemoveContainer" containerID="c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e"
Sep 12 03:19:47.549122 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:19:47.549106    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e\": container with ID starting with c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e not found: ID does not exist" containerID="c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e"
Sep 12 03:19:47.549169 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.549132    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e"} err="failed to get container status \"c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e\": rpc error: code = NotFound desc = could not find container \"c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e\": container with ID starting with c9355930dfeaf16250c05e43fb49a005b72bf38ae5234e88c8c9616a9dca616e not found: ID does not exist"
Sep 12 03:19:47.560110 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.560090    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"]
Sep 12 03:19:47.564090 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:47.564073    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b07066f0-kserve-ff6668899-wl7vf"]
Sep 12 03:19:48.245529 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:48.245497    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" path="/var/lib/kubelet/pods/2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2/volumes"
Sep 12 03:19:49.959612 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:49.959575    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"]
Sep 12 03:19:49.960170 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:49.960140    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" containerName="main"
Sep 12 03:19:49.960170 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:49.960160    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" containerName="main"
Sep 12 03:19:49.960332 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:49.960265    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8d31e1-c3e1-4f41-9f18-1b455e23e2e2" containerName="main"
Sep 12 03:19:49.964866 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:49.964837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:49.976522 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:49.976117    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"]
Sep 12 03:19:50.025143 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.025100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1cbfd-5223-4360-8912-20d3c23839d0-openshift-service-ca\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.025143 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.025138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltxq\" (UniqueName: \"kubernetes.io/projected/0fa1cbfd-5223-4360-8912-20d3c23839d0-kube-api-access-tltxq\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.025382 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.025214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fa1cbfd-5223-4360-8912-20d3c23839d0-maas-api-tls\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.025382 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.025247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa1cbfd-5223-4360-8912-20d3c23839d0-metrics-certs\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.126336 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.126298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1cbfd-5223-4360-8912-20d3c23839d0-openshift-service-ca\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.126537 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.126346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltxq\" (UniqueName: \"kubernetes.io/projected/0fa1cbfd-5223-4360-8912-20d3c23839d0-kube-api-access-tltxq\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.126537 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.126373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fa1cbfd-5223-4360-8912-20d3c23839d0-maas-api-tls\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.126537 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.126393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa1cbfd-5223-4360-8912-20d3c23839d0-metrics-certs\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.126965 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.126945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa1cbfd-5223-4360-8912-20d3c23839d0-openshift-service-ca\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.129063 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.129042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fa1cbfd-5223-4360-8912-20d3c23839d0-maas-api-tls\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.129155 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.129083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa1cbfd-5223-4360-8912-20d3c23839d0-metrics-certs\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.133865 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.133844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltxq\" (UniqueName: \"kubernetes.io/projected/0fa1cbfd-5223-4360-8912-20d3c23839d0-kube-api-access-tltxq\") pod \"maas-api-75df99c77c-75wnc\" (UID: \"0fa1cbfd-5223-4360-8912-20d3c23839d0\") " pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.286543 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.286437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:50.557582 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:50.557351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"]
Sep 12 03:19:50.560338 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:19:50.560311    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa1cbfd_5223_4360_8912_20d3c23839d0.slice/crio-833c44a8dbb99087df878e7f318bd4750c2a6f44b98d4835318d79159f7ebf2f WatchSource:0}: Error finding container 833c44a8dbb99087df878e7f318bd4750c2a6f44b98d4835318d79159f7ebf2f: Status 404 returned error can't find the container with id 833c44a8dbb99087df878e7f318bd4750c2a6f44b98d4835318d79159f7ebf2f
Sep 12 03:19:51.560964 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:51.560931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc" event={"ID":"0fa1cbfd-5223-4360-8912-20d3c23839d0","Type":"ContainerStarted","Data":"70eeedd561047d40b358d74e5965d44d883f6bef73ef41fd1d38b158e125e922"}
Sep 12 03:19:51.561388 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:51.560997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:51.561388 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:51.561023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc" event={"ID":"0fa1cbfd-5223-4360-8912-20d3c23839d0","Type":"ContainerStarted","Data":"833c44a8dbb99087df878e7f318bd4750c2a6f44b98d4835318d79159f7ebf2f"}
Sep 12 03:19:51.583652 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:51.583600    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc" podStartSLOduration=2.232429859 podStartE2EDuration="2.5835845s" totalImagesPullingTime="351.154641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 03:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:19:51.58174739 +0000 UTC m=+2609.864660972" watchObservedRunningTime="2026-09-12 03:19:51.5835845 +0000 UTC m=+2609.866498080"
Sep 12 03:19:57.571138 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:57.571109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75df99c77c-75wnc"
Sep 12 03:19:57.618959 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:57.618916    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"]
Sep 12 03:19:57.619140 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:57.619119    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" podUID="75f90316-0fdd-4202-9607-c259e0fa3eb7" containerName="maas-api" containerID="cri-o://e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6" gracePeriod=30
Sep 12 03:19:58.243156 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:58.243122    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" podUID="75f90316-0fdd-4202-9607-c259e0fa3eb7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": dial tcp 10.134.0.34:8443: connect: connection refused"
Sep 12 03:19:58.877834 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:58.877806    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 03:19:59.003102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.003017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-metrics-certs\") pod \"75f90316-0fdd-4202-9607-c259e0fa3eb7\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") "
Sep 12 03:19:59.003102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.003053    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76gtz\" (UniqueName: \"kubernetes.io/projected/75f90316-0fdd-4202-9607-c259e0fa3eb7-kube-api-access-76gtz\") pod \"75f90316-0fdd-4202-9607-c259e0fa3eb7\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") "
Sep 12 03:19:59.003102 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.003087    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-maas-api-tls\") pod \"75f90316-0fdd-4202-9607-c259e0fa3eb7\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") "
Sep 12 03:19:59.003401 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.003192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f90316-0fdd-4202-9607-c259e0fa3eb7-openshift-service-ca\") pod \"75f90316-0fdd-4202-9607-c259e0fa3eb7\" (UID: \"75f90316-0fdd-4202-9607-c259e0fa3eb7\") "
Sep 12 03:19:59.003619 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.003591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f90316-0fdd-4202-9607-c259e0fa3eb7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75f90316-0fdd-4202-9607-c259e0fa3eb7" (UID: "75f90316-0fdd-4202-9607-c259e0fa3eb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 03:19:59.005414 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.005388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "75f90316-0fdd-4202-9607-c259e0fa3eb7" (UID: "75f90316-0fdd-4202-9607-c259e0fa3eb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:19:59.005510 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.005424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75f90316-0fdd-4202-9607-c259e0fa3eb7" (UID: "75f90316-0fdd-4202-9607-c259e0fa3eb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 03:19:59.005571 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.005545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f90316-0fdd-4202-9607-c259e0fa3eb7-kube-api-access-76gtz" (OuterVolumeSpecName: "kube-api-access-76gtz") pod "75f90316-0fdd-4202-9607-c259e0fa3eb7" (UID: "75f90316-0fdd-4202-9607-c259e0fa3eb7"). InnerVolumeSpecName "kube-api-access-76gtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 03:19:59.104398 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.104364    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f90316-0fdd-4202-9607-c259e0fa3eb7-openshift-service-ca\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:59.104398 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.104391    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-metrics-certs\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:59.104398 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.104400    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gtz\" (UniqueName: \"kubernetes.io/projected/75f90316-0fdd-4202-9607-c259e0fa3eb7-kube-api-access-76gtz\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:59.104398 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.104409    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75f90316-0fdd-4202-9607-c259e0fa3eb7-maas-api-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 12 03:19:59.598520 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.598489    2579 generic.go:356] "Generic (PLEG): container finished" podID="75f90316-0fdd-4202-9607-c259e0fa3eb7" containerID="e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6" exitCode=0
Sep 12 03:19:59.598735 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.598532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"
Sep 12 03:19:59.598735 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.598573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" event={"ID":"75f90316-0fdd-4202-9607-c259e0fa3eb7","Type":"ContainerDied","Data":"e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6"}
Sep 12 03:19:59.598735 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.598621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl" event={"ID":"75f90316-0fdd-4202-9607-c259e0fa3eb7","Type":"ContainerDied","Data":"221306913e2a009b312b654558e2f09ca8098114c198f951455f4ce6db59a967"}
Sep 12 03:19:59.598735 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.598643    2579 scope.go:117] "RemoveContainer" containerID="e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6"
Sep 12 03:19:59.607826 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.607807    2579 scope.go:117] "RemoveContainer" containerID="e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6"
Sep 12 03:19:59.608116 ip-10-0-128-128 kubenswrapper[2579]: E0912 03:19:59.608100    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6\": container with ID starting with e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6 not found: ID does not exist" containerID="e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6"
Sep 12 03:19:59.608176 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.608125    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6"} err="failed to get container status \"e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6\": rpc error: code = NotFound desc = could not find container \"e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6\": container with ID starting with e88b6a8b33ac7b958f88f27605ffe0e76e6f002393a7ea1d08ab22ae8ae288c6 not found: ID does not exist"
Sep 12 03:19:59.619087 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.619061    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"]
Sep 12 03:19:59.622778 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:19:59.622758    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb8f5d89c-nqwpl"]
Sep 12 03:20:00.245068 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:00.245038    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f90316-0fdd-4202-9607-c259e0fa3eb7" path="/var/lib/kubelet/pods/75f90316-0fdd-4202-9607-c259e0fa3eb7/volumes"
Sep 12 03:20:24.451183 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:24.451148    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d5fbf76c7-fqmkz_eceb89ea-7b8d-4f58-82c7-e55025fa4650/authorino/0.log"
Sep 12 03:20:30.267286 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:30.267258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m7rxq_89d4480e-ab4e-473a-8c1b-21da5a18f9cc/manager/0.log"
Sep 12 03:20:30.389062 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:30.389034    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sv2vd_e9707ce9-097b-4407-8bb0-ecdaf7261264/manager/0.log"
Sep 12 03:20:30.518980 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:30.518902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5848789547-j6x7q_77c96c02-8728-4a76-a6da-b854d74c2666/manager/0.log"
Sep 12 03:20:30.639393 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:30.639365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vbm64_47b1a0db-5cb0-4190-9d58-0377565062e8/server/0.log"
Sep 12 03:20:30.904105 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:30.904078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-24wkz_b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf/copy-manifests/0.log"
Sep 12 03:20:30.915950 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:30.915928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-24wkz_b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf/manager/0.log"
Sep 12 03:20:31.538580 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:31.538554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75df99c77c-75wnc_0fa1cbfd-5223-4360-8912-20d3c23839d0/maas-api/0.log"
Sep 12 03:20:31.760001 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:31.759971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819700-n2sjg_52c660ca-99e5-4860-8ab0-834633c68b37/cleanup/0.log"
Sep 12 03:20:31.991972 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:31.991938    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-stjlq_c5644156-4463-4095-bb31-ece4310f815f/postgres/0.log"
Sep 12 03:20:32.808948 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:32.808917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d5fbf76c7-fqmkz_eceb89ea-7b8d-4f58-82c7-e55025fa4650/authorino/0.log"
Sep 12 03:20:32.929682 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:32.929656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dhdgg_75a3c81c-a0b3-44ef-83a3-3f36abade8c6/manager/0.log"
Sep 12 03:20:33.162203 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:33.162178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5gmn5_4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2/kuadrant-console-plugin/0.log"
Sep 12 03:20:33.535619 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:33.535537    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-rd7zb_d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e/manager/0.log"
Sep 12 03:20:34.142677 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:34.142647    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-7mdcm_7b38aa0d-1f5c-4f31-945b-fb9eec484e4d/kube-auth-proxy/0.log"
Sep 12 03:20:34.380329 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:34.380298    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/1.log"
Sep 12 03:20:34.628684 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:34.628649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l7qnf_0b574436-4216-4db3-8a6d-f53d83910cf7/payload-processing/0.log"
Sep 12 03:20:35.347483 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:35.347430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-5h5mm_5ae59272-69af-4bcc-89f8-9373e32bcda5/main/0.log"
Sep 12 03:20:35.712048 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:35.712020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cv2kn_1a29b208-1fa2-4601-a6dd-3bf801054398/main/0.log"
Sep 12 03:20:41.728657 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:41.728627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:20:41.829801 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:41.829773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log" limit=1024
Sep 12 03:20:44.527128 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:44.527099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jwttl_2a32de6d-b07f-4d06-ac75-8158fcaaa3a2/global-pull-secret-syncer/0.log"
Sep 12 03:20:44.651609 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:44.651579    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cggzr_f2641236-d461-4b49-a03d-5f500d3fb573/konnectivity-agent/0.log"
Sep 12 03:20:44.742635 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:44.742608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-128.ec2.internal_0b29c01eae268a79203b6e1217a43813/haproxy/0.log"
Sep 12 03:20:49.696305 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:49.696275    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d5fbf76c7-fqmkz_eceb89ea-7b8d-4f58-82c7-e55025fa4650/authorino/0.log"
Sep 12 03:20:49.733044 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:49.733013    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dhdgg_75a3c81c-a0b3-44ef-83a3-3f36abade8c6/manager/0.log"
Sep 12 03:20:49.797378 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:49.797351    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5gmn5_4afb5ac9-4a91-40b3-8ab6-dcc1b04d36e2/kuadrant-console-plugin/0.log"
Sep 12 03:20:49.941654 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:49.941625    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-rd7zb_d2f5d4c4-c13b-41c3-8a7c-164094fc4f8e/manager/0.log"
Sep 12 03:20:51.858103 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:51.858071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7slnk_79c2dd42-3472-444a-8710-de01329994af/node-exporter/0.log"
Sep 12 03:20:51.876854 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:51.876826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7slnk_79c2dd42-3472-444a-8710-de01329994af/kube-rbac-proxy/0.log"
Sep 12 03:20:51.898603 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:51.898575    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7slnk_79c2dd42-3472-444a-8710-de01329994af/init-textfile/0.log"
Sep 12 03:20:55.851216 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.851176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"]
Sep 12 03:20:55.851668 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.851577    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f90316-0fdd-4202-9607-c259e0fa3eb7" containerName="maas-api"
Sep 12 03:20:55.851668 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.851591    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f90316-0fdd-4202-9607-c259e0fa3eb7" containerName="maas-api"
Sep 12 03:20:55.851776 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.851678    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f90316-0fdd-4202-9607-c259e0fa3eb7" containerName="maas-api"
Sep 12 03:20:55.854944 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.854924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:55.857427 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.857403    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vchk2\"/\"openshift-service-ca.crt\""
Sep 12 03:20:55.857576 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.857428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vchk2\"/\"kube-root-ca.crt\""
Sep 12 03:20:55.857576 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.857495    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vchk2\"/\"default-dockercfg-brcb4\""
Sep 12 03:20:55.864857 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.864822    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"]
Sep 12 03:20:55.999197 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.999163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-podres\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:55.999349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.999208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-sys\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:55.999349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.999307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-lib-modules\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:55.999349 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.999346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjvmz\" (UniqueName: \"kubernetes.io/projected/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-kube-api-access-bjvmz\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:55.999531 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:55.999364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-proc\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.099977 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.099941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjvmz\" (UniqueName: \"kubernetes.io/projected/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-kube-api-access-bjvmz\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.099977 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.099976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-proc\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-podres\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-sys\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-lib-modules\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-proc\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-sys\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-podres\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.100210 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.100207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-lib-modules\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.108479 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.108424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjvmz\" (UniqueName: \"kubernetes.io/projected/f1cb8313-ac99-4778-bc6d-52aa3e6d4f15-kube-api-access-bjvmz\") pod \"perf-node-gather-daemonset-q7442\" (UID: \"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15\") " pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.166042 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.166001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.436363 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.436339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"]
Sep 12 03:20:56.438539 ip-10-0-128-128 kubenswrapper[2579]: W0912 03:20:56.438509    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1cb8313_ac99_4778_bc6d_52aa3e6d4f15.slice/crio-45f8179b784c8ee69bd85926e167cf42a99dd2f3fdf4818db8770db670efbc36 WatchSource:0}: Error finding container 45f8179b784c8ee69bd85926e167cf42a99dd2f3fdf4818db8770db670efbc36: Status 404 returned error can't find the container with id 45f8179b784c8ee69bd85926e167cf42a99dd2f3fdf4818db8770db670efbc36
Sep 12 03:20:56.861367 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.861330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442" event={"ID":"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15","Type":"ContainerStarted","Data":"f4a3abdb81ae202c8bad02ec217bd1b29c0714c763f3ef5a1ebf754dee39c404"}
Sep 12 03:20:56.861789 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.861392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442" event={"ID":"f1cb8313-ac99-4778-bc6d-52aa3e6d4f15","Type":"ContainerStarted","Data":"45f8179b784c8ee69bd85926e167cf42a99dd2f3fdf4818db8770db670efbc36"}
Sep 12 03:20:56.861789 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.861439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:20:56.877739 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:56.877685    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442" podStartSLOduration=1.877666333 podStartE2EDuration="1.877666333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 03:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 03:20:56.876098398 +0000 UTC m=+2675.159011977" watchObservedRunningTime="2026-09-12 03:20:56.877666333 +0000 UTC m=+2675.160579914"
Sep 12 03:20:57.613622 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:57.613597    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgkgx_e80ade3b-dcdf-4c97-96ce-4d14057a7b64/dns/0.log"
Sep 12 03:20:57.633174 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:57.633146    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgkgx_e80ade3b-dcdf-4c97-96ce-4d14057a7b64/kube-rbac-proxy/0.log"
Sep 12 03:20:57.699607 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:57.699576    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kf2vn_ef2c7eeb-c2d2-40cb-a9ad-068694595411/dns-node-resolver/0.log"
Sep 12 03:20:58.218346 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:58.218316    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6fdf8bffd8-6278h_cdd3e22f-3a82-4f78-9e5a-83ca725b240d/registry/0.log"
Sep 12 03:20:58.265392 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:58.265362    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jrpjs_1ed4de21-cc73-4384-8dd2-c4ed4710e18a/node-ca/0.log"
Sep 12 03:20:59.641319 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:59.641286    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-7mdcm_7b38aa0d-1f5c-4f31-945b-fb9eec484e4d/kube-auth-proxy/0.log"
Sep 12 03:20:59.694371 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:59.694341    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/0.log"
Sep 12 03:20:59.696889 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:59.696867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/1.log"
Sep 12 03:20:59.752887 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:20:59.752858    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l7qnf_0b574436-4216-4db3-8a6d-f53d83910cf7/payload-processing/0.log"
Sep 12 03:21:00.717423 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:00.717395    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jskpx_08ba0220-aa22-4bbe-8901-8f2bf3d70320/serve-healthcheck-canary/0.log"
Sep 12 03:21:01.386600 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:01.386540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdw5q_c285a525-1cfe-4e6a-9f24-072e1788a672/kube-rbac-proxy/0.log"
Sep 12 03:21:01.407405 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:01.407374    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdw5q_c285a525-1cfe-4e6a-9f24-072e1788a672/exporter/0.log"
Sep 12 03:21:01.428719 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:01.428694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdw5q_c285a525-1cfe-4e6a-9f24-072e1788a672/extractor/0.log"
Sep 12 03:21:02.876135 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:02.876110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vchk2/perf-node-gather-daemonset-q7442"
Sep 12 03:21:03.557001 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:03.556974    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m7rxq_89d4480e-ab4e-473a-8c1b-21da5a18f9cc/manager/0.log"
Sep 12 03:21:03.811513 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:03.811403    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sv2vd_e9707ce9-097b-4407-8bb0-ecdaf7261264/manager/0.log"
Sep 12 03:21:03.856190 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:03.856161    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5848789547-j6x7q_77c96c02-8728-4a76-a6da-b854d74c2666/manager/0.log"
Sep 12 03:21:03.880928 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:03.880903    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vbm64_47b1a0db-5cb0-4190-9d58-0377565062e8/server/0.log"
Sep 12 03:21:04.040759 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:04.040729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-24wkz_b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf/manager/0.log"
Sep 12 03:21:04.056226 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:04.056204    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774995dcdd-24wkz_b44d2542-e2ce-4d5c-baaa-e9f6e308ffaf/copy-manifests/0.log"
Sep 12 03:21:07.433491 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:07.433438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-7mdcm_7b38aa0d-1f5c-4f31-945b-fb9eec484e4d/kube-auth-proxy/0.log"
Sep 12 03:21:07.469015 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:07.468983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/0.log"
Sep 12 03:21:07.471540 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:07.471519    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/1.log"
Sep 12 03:21:07.517891 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:07.517862    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l7qnf_0b574436-4216-4db3-8a6d-f53d83910cf7/payload-processing/0.log"
Sep 12 03:21:09.608076 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:09.608048    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84dbf54759-7mdcm_7b38aa0d-1f5c-4f31-945b-fb9eec484e4d/kube-auth-proxy/0.log"
Sep 12 03:21:09.642570 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:09.642535    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/0.log"
Sep 12 03:21:09.644492 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:09.644472    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5j9c8_fd97812e-9629-4da1-a7e5-dccfcb749b45/istio-proxy/1.log"
Sep 12 03:21:09.686915 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:09.686889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-l7qnf_0b574436-4216-4db3-8a6d-f53d83910cf7/payload-processing/0.log"
Sep 12 03:21:12.770007 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.769980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/kube-multus-additional-cni-plugins/0.log"
Sep 12 03:21:12.790305 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.790279    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/egress-router-binary-copy/0.log"
Sep 12 03:21:12.809201 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.809175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/cni-plugins/0.log"
Sep 12 03:21:12.827913 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.827850    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/bond-cni-plugin/0.log"
Sep 12 03:21:12.847337 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.847316    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/routeoverride-cni/0.log"
Sep 12 03:21:12.869433 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.869407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/whereabouts-cni-bincopy/0.log"
Sep 12 03:21:12.888844 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:12.888809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nczqn_3af0f662-36c5-4666-a957-0ab576a736bc/whereabouts-cni/0.log"
Sep 12 03:21:13.026260 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:13.026178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xn4b5_93a70706-fe88-4581-a774-f851f980d9b0/kube-multus/0.log"
Sep 12 03:21:13.086743 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:13.086718    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nj5d4_defb704d-5be6-4ab6-b3f0-e381e9dbb3aa/network-metrics-daemon/0.log"
Sep 12 03:21:13.104980 ip-10-0-128-128 kubenswrapper[2579]: I0912 03:21:13.104956    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nj5d4_defb704d-5be6-4ab6-b3f0-e381e9dbb3aa/kube-rbac-proxy/0.log"